Search Results: "tbm"

29 June 2023

C.J. Collier: Converting a windows install to a libvirt VM

Reduce the size of your c: partition to the smallest it can be and then turn off windows with the understanding that you will never boot this system on the iron ever again.
Boot into a netinst installer image (no GUI). hold alt and press left arrow a few times until you get to a prompt to press enter. Press enter. In this example /dev/sda is your windows disk which contains the c: partition
and /dev/disk/by-id/usb0 is the USB-3 attached SATA controller that you have your SSD attached to (please find an example attached). This SSD should be equal to or larger than the windows disk for best compatability. A photo of a USB-3 attached SATA controller To find the literal path names of your detected drives you can run fdisk -l. Pay attention to the names of the partitions and the sizes of the drives to help determine which is which. Once you have a shell in the netinst installer, you should maybe be able to run a command like the following. This will duplicate the disk located at if (in file) to the disk located at of (out file) while showing progress as the status.
dd if=/dev/sda of=/dev/disk/by-id/usb0 status=progress
If you confirm that dd is available on the netinst image and the previous command runs successfully, test that your windows partition is visible in the new disk s partition table. The start block of the windows partition on each should match, as should the partition size.
fdisk -l /dev/disk/by-id/usb0
fdisk -l /dev/sda
If the output from the first is the same as the output from the second, then you are probably safe to proceed. Once you confirm that you have made and tested a full copy of the blocks from your windows drive saved on your usb disk, nuke your windows partition table from orbit.
dd if=/dev/zero of=/dev/sda bs=1M count=42
You can press alt-f1 to return to the Debian installer now. Follow the instructions to install Debian. Don t forget to remove all attached USB drives. Once you install Debian, press ctrl-alt-f3 to get a root shell. Add your user to the sudoers group:
# adduser cjac sudoers
log out
# exit
log in as your user and confirm that you have sudo
$ sudo ls
Don t forget to read the spider man advice enter your password you ll need to install virt-manager. I think this should help:
$ sudo apt-get install virt-manager libvirt-daemon-driver-qemu qemu-system-x86
insert the USB drive. You can now create a qcow2 file for your virtual machine.
$ sudo qemu-img convert -O qcow2 \
/dev/disk/by-id/usb0 \
/var/lib/libvirt/images/windows.qcow2
I personally create a volume group called /dev/vg00 for the stuff I want to run raw and instead of converting to qcow2 like all of the other users do, I instead write it to a new logical volume.
sudo lvcreate /dev/vg00 -n windows -L 42G # or however large your drive was
sudo dd if=/dev/disk/by-id/usb0 of=/dev/vg00/windows status=progress
Now that you ve got the qcow2 file created, press alt-left until you return to your GDM session. The apt-get install command above installed virt-manager, so log in to your system if you haven t already and open up gnome-terminal by pressing the windows key or moving your mouse/gesture to the top left of your screen. Type in gnome-terminal and either press enter or click/tap on the icon. I like to run this full screen so that I feel like I m in a space ship. If you like to feel like you re in a spaceship, too, press F11. You can start virt-manager from this shell or you can press the windows key and type in virt-manager and press enter. You ll want the shell to run commands such as virsh console windows or virsh list When virt-manager starts, right click on QEMU/KVM and select New.
In the New VM window, select Import existing disk image
When prompted for the path to the image, use the one we created with sudo qemu-img convert above.
Select the version of Windows you want.
Select memory and CPUs to allocate to the VM.
Tick the Customize configuration before install box
If you re prompted to enable the default network, do so now.
The default hardware layout should probably suffice. Get it as close to the underlying hardware as it is convenient to do. But Windows is pretty lenient these days about virtualizing licensed windows instances so long as they re not running in more than one place at a time. Good luck! Leave comments if you have questions.

9 February 2023

Jonathan McDowell: Building a read-only Debian root setup: Part 2

This is the second part of how I build a read-only root setup for my router. You might want to read part 1 first, which covers the initial boot and general overview of how I tie the pieces together. This post will describe how I build the squashfs image that forms the main filesystem. Most of the build is driven from a script, make-router, which I ll dissect below. It s highly tailored to my needs, and this is a fairly lengthy post, but hopefully the steps I describe prove useful to anyone trying to do something similar.
Breakdown of make-router
#!/bin/bash
# Either rb3011 (arm) or rb5009 (arm64)
#HOSTNAME="rb3011"
HOSTNAME="rb5009"
if [ "x$ HOSTNAME " == "xrb3011" ]; then
	ARCH=armhf
elif [ "x$ HOSTNAME " == "xrb5009" ]; then
	ARCH=arm64
else
	echo "Unknown host: $ HOSTNAME "
	exit 1
fi

It s a bash script, and I allow building for either my RB3011 or RB5009, which means a different architecture (32 vs 64 bit). I run this script on my Pi 4 which means I don t have to mess about with QemuUserEmulation.
BASE_DIR=$(dirname $0)
IMAGE_FILE=$(mktemp --tmpdir router.$ ARCH .XXXXXXXXXX.img)
MOUNT_POINT=$(mktemp -p /mnt -d router.$ ARCH .XXXXXXXXXX)
# Build and mount an ext4 image file to put the root file system in
dd if=/dev/zero bs=1 count=0 seek=1G of=$ IMAGE_FILE 
mkfs -t ext4 $ IMAGE_FILE 
mount -o loop $ IMAGE_FILE  $ MOUNT_POINT 

I build the image in a loopback ext4 file on tmpfs (my Pi4 is the 8G model), which makes things a bit faster.
# Add dpkg excludes
mkdir -p $ MOUNT_POINT /etc/dpkg/dpkg.cfg.d/
cat <<EOF > $ MOUNT_POINT /etc/dpkg/dpkg.cfg.d/path-excludes
# Exclude docs
path-exclude=/usr/share/doc/*
# Only locale we want is English
path-exclude=/usr/share/locale/*
path-include=/usr/share/locale/en*/*
path-include=/usr/share/locale/locale.alias
# No man pages
path-exclude=/usr/share/man/*
EOF

Create a dpkg excludes config to drop docs, man pages and most locales before we even start the bootstrap.
# Setup fstab + mtab
echo "# Empty fstab as root is pre-mounted" > $ MOUNT_POINT /etc/fstab
ln -s ../proc/self/mounts $ MOUNT_POINT /etc/mtab
# Setup hostname
echo $ HOSTNAME  > $ MOUNT_POINT /etc/hostname
# Add the root SSH keys
mkdir -p $ MOUNT_POINT /root/.ssh/
cat <<EOF > $ MOUNT_POINT /root/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8NkUeVdsVdegS+JT9qwFwiHEgcC9sBwnv6RjpH6I4d3im4LOaPOatzneMTZlH8Gird+H4nzluciBr63hxmcFjZVW7dl6mxlNX2t/wKvV0loxtEmHMoI7VMCnrWD0PyvwJ8qqNu9cANoYriZRhRCsBi27qPNvI741zEpXN8QQs7D3sfe4GSft9yQplfJkSldN+2qJHvd0AHKxRdD+XTxv1Ot26+ZoF3MJ9MqtK+FS+fD9/ESLxMlOpHD7ltvCRol3u7YoaUo2HJ+u31l0uwPZTqkPNS9fkmeCYEE0oXlwvUTLIbMnLbc7NKiLgniG8XaT0RYHtOnoc2l2UnTvH5qsQ== noodles@earth.li
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQb9+qFemcwKhey3+eTh5lxp+3sgZXW2HQQEZMt9hPvVXk+MiiNMx9WUzxPJnwXqlmmVdKsq+AvjA0i505Pp8fIj5DdUBpSqpLghmzpnGuob7SSwXYj+352hjD52UC4S0KMKbIaUpklADgsCbtzhYYc4WoO8F7kK63tS5qa1XSZwwRwPbYOWBcNocfr9oXCVWD9ismO8Y0l75G6EyW8UmwYAohDaV83pvJxQerYyYXBGZGY8FNjqVoOGMRBTUcLj/QTo0CDQvMtsEoWeCd0xKLZ3gjiH3UrknkaPra557/TWymQ8Oh15aPFTr5FvKgAlmZaaM0tP71SOGmx7GpCsP4jZD1Xj/7QMTAkLXb+Ou6yUOVM9J4qebdnmF2RGbf1bwo7xSIX6gAYaYgdnppuxqZX1wyAy+A2Hie4tUjMHKJ6OoFwBsV1sl+3FobrPn6IuulRCzsq2aLqLey+PHxuNAYdSKo7nIDB3qCCPwHlDK52WooSuuMidX4ujTUw7LDTia9FxAawudblxbrvfTbg3DsiDBAOAIdBV37HOAKu3VmvYSPyqT80DEy8KFmUpCEau59DID9VERkG6PWPVMiQnqgW2Agn1miOBZeIQV8PFjenAySxjzrNfb4VY/i/kK9nIhXn92CAu4nl6D+VUlw+IpQ8PZlWlvVxAtLonpjxr9OTw== noodles@yubikey
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0I8UHj4IpfqUcGE4cTvLB0d2xmATSUzqtxW6ZhGbZxvQDKJesVW6HunrJ4NFTQuQJYgOXY/o82qBpkEKqaJMEFHTCjcaj3M6DIaxpiRfQfs0nhtzDB6zPiZn9Suxb0s5Qr4sTWd6iI9da72z3hp9QHNAu4vpa4MSNE+al3UfUisUf4l8TaBYKwQcduCE0z2n2FTi3QzmlkOgH4MgyqBBEaqx1tq7Zcln0P0TYZXFtrxVyoqBBIoIEqYxmFIQP887W50wQka95dBGqjtV+d8IbrQ4pB55qTxMd91L+F8n8A6nhQe7DckjS0Xdla52b9RXNXoobhtvx9K2prisagsHT noodles@cup
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK6iGog3WbNhrmrkglNjVO8/B6m7mN6q1tMm1sXjLxQa+F86ETTLiXNeFQVKCHYrk8f7hK0d2uxwgj6Ixy9k0Cw= noodles@sevai
EOF

Setup fstab, the hostname and SSH keys for root.
# Bootstrap our install
debootstrap \
	--arch=$ ARCH  \
	--include=collectd-core,conntrack,dnsmasq,ethtool,iperf3,kexec-tools,mosquitto,mtd-utils,mtr-tiny,ppp,tcpdump,rng-tools5,ssh,watchdog,wget \
	--exclude=dmidecode,isc-dhcp-client,isc-dhcp-common,makedev,nano \
	bullseye $ MOUNT_POINT  https://deb.debian.org/debian/

Actually do the debootstrap step, including a bunch of extra packages that we want.
# Install mqtt-arp
cp $ BASE_DIR /debs/mqtt-arp_1_$ ARCH .deb $ MOUNT_POINT /tmp
chroot $ MOUNT_POINT  dpkg -i /tmp/mqtt-arp_1_$ ARCH .deb
rm $ MOUNT_POINT /tmp/mqtt-arp_1_$ ARCH .deb
# Frob the mqtt-arp config so it starts after mosquitto
sed -i -e 's/After=.*/After=mosquitto.service/' $ MOUNT_POINT /lib/systemd/system/mqtt-arp.service

I haven t uploaded mqtt-arp to Debian, so I install a locally built package, and ensure it starts after mosquitto (the MQTT broker), given they re running on the same host.
# Frob watchdog so it starts earlier than multi-user
sed -i -e 's/After=.*/After=basic.target/' $ MOUNT_POINT /lib/systemd/system/watchdog.service
# Make sure the watchdog is poking the device file
sed -i -e 's/^#watchdog-device/watchdog-device/' $ MOUNT_POINT /etc/watchdog.conf

watchdog timeouts were particularly an issue on the RB3011, where the default timeout didn t give enough time to reach multiuser mode before it would reset the router. Not helpful, so alter the config to start it earlier (and make sure it s configured to actually kick the device file).
# Clean up docs + locales
rm -r $ MOUNT_POINT /usr/share/doc/*
rm -r $ MOUNT_POINT /usr/share/man/*
for dir in $ MOUNT_POINT /usr/share/locale/*/; do
	if [ "$ dir " != "$ MOUNT_POINT /usr/share/locale/en/" ]; then
		rm -r $ dir 
	fi
done

Clean up any docs etc that ended up installed.
# Set root password to root
echo "root:root"   chroot $ MOUNT_POINT  chpasswd

The only login method is ssh key to the root account though I suppose this allows for someone to execute a privilege escalation from a daemon user so I should probably randomise this. Does need to be known though so it s possible to login via the serial console for debugging.
# Add security to sources.list + update
echo "deb https://security.debian.org/debian-security bullseye-security main" >> $ MOUNT_POINT /etc/apt/sources.list
chroot $ MOUNT_POINT  apt update
chroot $ MOUNT_POINT  apt -y full-upgrade
chroot $ MOUNT_POINT  apt clean
# Cleanup the APT lists
rm $ MOUNT_POINT /var/lib/apt/lists/www.*
rm $ MOUNT_POINT /var/lib/apt/lists/security.*

Pull in any security updates, then clean out the APT lists rather than polluting the image with them.
# Disable the daily APT timer
rm $ MOUNT_POINT /etc/systemd/system/timers.target.wants/apt-daily.timer
# Disable daily dpkg backup
cat <<EOF > $ MOUNT_POINT /etc/cron.daily/dpkg
#!/bin/sh
# Don't do the daily dpkg backup
exit 0
EOF
# We don't want a persistent systemd journal
rmdir $ MOUNT_POINT /var/log/journal

None of these make sense on a router.
# Enable nftables
ln -s /lib/systemd/system/nftables.service \
	$ MOUNT_POINT /etc/systemd/system/sysinit.target.wants/nftables.service

Ensure we have firewalling enabled automatically.
# Add systemd-coredump + systemd-timesync user / group
echo "systemd-timesync:x:998:" >> $ MOUNT_POINT /etc/group
echo "systemd-coredump:x:999:" >> $ MOUNT_POINT /etc/group
echo "systemd-timesync:!*::" >> $ MOUNT_POINT /etc/gshadow
echo "systemd-coredump:!*::" >> $ MOUNT_POINT /etc/gshadow
echo "systemd-timesync:x:998:998:systemd Time Synchronization:/:/usr/sbin/nologin" >> $ MOUNT_POINT /etc/passwd
echo "systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin" >> $ MOUNT_POINT /etc/passwd
echo "systemd-timesync:!*:47358::::::" >> $ MOUNT_POINT /etc/shadow
echo "systemd-coredump:!*:47358::::::" >> $ MOUNT_POINT /etc/shadow
# Create /etc/.pwd.lock, otherwise it'll end up in the overlay
touch $ MOUNT_POINT /etc/.pwd.lock
chmod 600 $ MOUNT_POINT /etc/.pwd.lock

Create a number of users that will otherwise get created at boot, and a lock file that will otherwise get created anyway.
# Copy config files
cp --recursive --preserve=mode,timestamps $ BASE_DIR /etc/* $ MOUNT_POINT /etc/
cp --recursive --preserve=mode,timestamps $ BASE_DIR /etc-$ ARCH /* $ MOUNT_POINT /etc/
chroot $ MOUNT_POINT  chown mosquitto /etc/mosquitto/mosquitto.users
chroot $ MOUNT_POINT  chown mosquitto /etc/ssl/mqtt.home.key

There are config files that are easier to replace wholesale, some of which are specific to the hardware (e.g. related to network interfaces). See below for some more details.
# Build symlinks into flash for boot / modules
ln -s /mnt/flash/lib/modules $ MOUNT_POINT /lib/modules
rmdir $ MOUNT_POINT /boot
ln -s /mnt/flash/boot $ MOUNT_POINT /boot

The kernel + its modules live outside the squashfs image, on the USB flash drive that the image lives on. That makes for easier kernel upgrades.
# Put our git revision into os-release
echo -n "GIT_VERSION=" >> $ MOUNT_POINT /etc/os-release
(cd $ BASE_DIR  ; git describe --tags) >> $ MOUNT_POINT /etc/os-release

Always helpful to be able to check the image itself for what it was built from.
# Add some stuff to root's .bashrc
cat << EOF >> $ MOUNT_POINT /root/.bashrc
alias ls='ls -F --color=auto'
eval "\$(dircolors)"
case "\$TERM" in
xterm* rxvt*)
	PS1="\\[\\e]0;\\u@\\h: \\w\a\\]\$PS1"
	;;
*)
	;;
esac
EOF

Just some niceties for when I do end up logging in.
# Build the squashfs
mksquashfs $ MOUNT_POINT  /tmp/router.$ ARCH .squashfs \
	-comp xz

Actually build the squashfs image.
# Save the installed package list off
chroot $ MOUNT_POINT  dpkg --get-selections > /tmp/wip-installed-packages

Save off the installed package list. This was particularly useful when trying to replicate the existing router setup and making sure I had all the important packages installed. It doesn t really serve a purpose now.
In terms of the config files I copy into /etc, shared across both routers are the following:
Breakdown of shared config
  • apt config (disable recommends, periodic updates):
    • apt/apt.conf.d/10periodic, apt/apt.conf.d/local-recommends
  • Adding a default, empty, locale:
    • default/locale
  • DNS/DHCP:
    • dnsmasq.conf, dnsmasq.d/dhcp-ranges, dnsmasq.d/static-ips
    • hosts, resolv.conf
  • Enabling IP forwarding:
    • sysctl.conf
  • Logs related:
    • logrotate.conf, rsyslog.conf
  • MQTT related:
    • mosquitto/mosquitto.users, mosquitto/conf.d/ssl.conf, mosquitto/conf.d/users.conf, mosquitto/mosquitto.acl, mosquitto/mosquitto.conf
    • mqtt-arp.conf
    • ssl/lets-encrypt-r3.crt, ssl/mqtt.home.key, ssl/mqtt.home.crt
  • PPP configuration:
    • ppp/ip-up.d/0000usepeerdns, ppp/ipv6-up.d/defaultroute, ppp/pap-secrets, ppp/chap-secrets
    • network/interfaces.d/pppoe-wan
The router specific config is mostly related to networking:
Breakdown of router specific config
  • Firewalling:
    • nftables.conf
  • Interfaces:
    • dnsmasq.d/interfaces
    • network/interfaces.d/eth0, network/interfaces.d/p1, network/interfaces.d/p2, network/interfaces.d/p7, network/interfaces.d/p8
  • PPP config (network interface piece):
    • ppp/peers/aquiss
  • SSH keys:
    • ssh/ssh_host_ecdsa_key, ssh/ssh_host_ed25519_key, ssh/ssh_host_rsa_key, ssh/ssh_host_ecdsa_key.pub, ssh/ssh_host_ed25519_key.pub, ssh/ssh_host_rsa_key.pub
  • Monitoring:
    • collectd/collectd.conf, collectd/collectd.conf.d/network.conf

28 April 2021

Martin Michlmayr: Research on FOSS foundations

I worked on research on FOSS foundations and published two reports: Growing Open Source Projects with a Stable Foundation This primer covers non-technical aspects that the majority of projects will have to consider at some point. It also explains how FOSS foundations can help projects grow and succeed. This primer explains: You can download Growing Open Source Projects with a Stable Foundation. Research report The research report describes the findings of the research and aims to help understand the operations and challenges FOSS foundations face. This report covers topics such as: You can download the research report. Acknowledgments This research was sponsored by Ford Foundation and Alfred P. Sloan Foundation. The research was part of their Critical Digital Infrastructure Research initiative, which investigates the role of open source in digital infrastructure.

15 April 2021

Martin Michlmayr: ledger2beancount 2.6 released

I released version 2.6 of ledger2beancount, a ledger to beancount converter. Here are the changes in 2.6: Thanks to Alexander Baier, Daniele Nicolodi, and GitHub users bratekarate, faaafo and mefromthepast for various bug reports and other input. Thanks to Dennis Lee for adding a Dockerfile and to Vinod Kurup for fixing a bug. Thanks to Stefano Zacchiroli for testing. You can get ledger2beancount from GitHub.

13 November 2020

Martin Michlmayr: beancount2ledger 1.3 released

I released version 1.3 of beancount2ledger, the beancount to ledger converter that was moved from bean-report ledger into a standalone tool. You can get beancount2ledger from GitHub or via pip install. Here are the changes in 1.3:

3 November 2020

Martin Michlmayr: ledger2beancount 2.5 released

I released version 2.5 of ledger2beancount, a ledger to beancount converter. Here are the changes in 2.5: Thanks to input from Remco R nders, Yuri Khan, and Thierry. Thanks to Stefano Zacchiroli and Kirill Goncharov for testing my changes. You can get ledger2beancount from GitHub

27 July 2020

Martin Michlmayr: ledger2beancount 2.4 released

I released version 2.4 of ledger2beancount, a ledger to beancount converter. There are two notable changes in this release:
  1. I fixed two regressions introduced in the last release. Sorry about the breakage!
  2. I improved support for hledger. I believe all syntax differences in hledger are supported now.
Here are the changes in 2.4: Thanks to Kirill Goncharov for pointing out one regressions, to Taylor R Campbell for for a patch, to Stefano Zacchiroli for some input, and finally to Simon Michael for input on hledger! You can get ledger2beancount from GitHub

24 July 2020

Martin Michlmayr: beancount2ledger 1.1 released

Martin Blais recently announced that he'd like to re-organize the beancount code and split out some functionality into separate projects, including the beancount to ledger/hledger conversion code previously provided by bean-report. I agreed to take on the maintenance of this code and I've now released beancount2ledger, a beancount to ledger/hledger converter. You can install beancount2ledger with pip:
pip3 install beancount2ledger
Please report issues to the GitHub tracker. There are a number of outstanding issues I'll fix soon, but please report any other issues you encounter. Note that I'm not very familiar with hledger. I intend to sync up with hledger author Simon Michael soon, but please file an issue if you notice any problems with the hledger conversion. Version 1.1 contains a number of fixes compared to the latest code in bean-report: 1.1 (2020-07-24) 1.0 (2020-07-22)

26 June 2020

Martin Michlmayr: ledger2beancount 2.3 released

I released version 2.3 of ledger2beancount, a ledger to beancount converter. There are three notable changes with this release:
  1. Performance has significantly improved. One large, real-world test case has gone from around 160 seconds to 33 seconds. A smaller test case has gone from 11 seconds to ~3.5 seconds.
  2. The documentation is available online now (via Read the Docs).
  3. The repository has moved to the beancount GitHub organization.
Here are the changes in 2.3: Thanks to Colin Dean for some feedback. Thanks to Stefano Zacchiroli for prompting me into investigating performance issues (and thanks to the developers of the Devel::NYTProf profiler). You can get ledger2beancount from GitHub

11 June 2020

C.J. Adams-Collier: Recovering videos from DV tapes with Canon ZR80

I am recovering some tapes from back in the day that some of you may enjoy. Here is a log of the process so that maybe you can recover some of your own DV tapes. Seems to work well in modern Debian. To attach to the camcorder, I used a PCI-e card that has an old firewire port and some ASIC on board. The PCI card came up and loaded the correct kernel drivers. Here is a search link so that you can buy a similar card. cjac@server0:~$ sudo lspci grep 1394
b2:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEE
E 1394 OHCI Controller (rev 46) cjac@server0:~$ sudo lsmod grep -i firewire
firewire_ohci 45056 0
firewire_core 81920 7 firewire_ohci
crc_itu_t 16384 1 firewire_core The dvgrab program is available on Debian under the dvgrab package.
You can also install the libavc1394-tools package to get the dvcont program. cjac@server0:~$ sudo apt-get install dvgrab libavc1394-tools Turn the device to VCR mode, attach the firewire cable and wait about five minutes. Have you watered the cat today? cjac@server0:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 10 (buster)
Release: 10
Codename: buster
cjac@server0:~$ uname -r
5.2.0-0.bpo.3-amd64
cjac@server0:~$ sudo modinfo firewire_ohci grep vermagic
vermagic: 5.2.0-0.bpo.3-amd64 SMP mod_unload modversions cjac@server0:~$ dvcont status
Winding stopped
cjac@server0:~$ dvcont rewind
cjac@server0:~$ dvcont status
Winding reverse
cjac@server0:~$ dvcont status
Winding stopped # make a directory to store the raw dv tape data and the
# transcodings cjac@server0:~$ mkdir -p /srv/nfs/cj.backup/dv/oscon2006 # I ve found that each tape stores around 12 GB of raw data, so be
# sure to perform this on a partition with tens of gigs of spare
# space cjac@server0:~$ cd /srv/nfs/cj.backup/dv/oscon2006
cjac@server0:/srv/nfs/cj.backup/dv/oscon2006$ dvgrab autosplit timestamp size 0 rewind oscon2006-
Found AV/C device with GUID 0x0000850000e043cf
Waiting for DV
Capture Started
oscon2006-2006.07.26_12-37-44.dv : 266.30 MiB 2327 frames timecode 00:01:17.26 date 2006.07.26 12:39:01
oscon2006-2006.07.26_12-40-59.dv : 816.76 MiB 7137 frames timecode 00:05:16.01 date 2006.07.26 12:44:57
oscon2006-2006.07.26_12-45-06.dv : 8420.56 MiB 73580 frames timecode 00:46:11.05 date 2006.07.26 13:26:01
oscon2006-2006.07.26_13-32-08.dv : 2961.27 MiB 25876 frames timecode 00:00:00.00 date 2020.06.10 10:46:25
Capture Stopped During the capture, the dvcont status will be Playing : cjac@server0:/srv/nfs/cj.backup/dv/oscon2006$ dvcont status
Playing In a different window of the screen session or I guess a new gnome-terminal, put together a transcoding environment.
libx264-155 cjac@server0:/srv/nfs/cj.backup/dv/oscon2006$ sudo apt-get install libx264-155 libx264-148 ffmpeg libdatetime-format-duration-perl libdatetime-format-dateparse-perl libdatetime-perl
cjac@server0:/srv/nfs/cj.backup/dv/oscon2006$ wget https://raw.githubusercontent.com/cjac/dvscripts/master/transcode.pl && chmod u+x transcode.pl
# review transcode.pl, change $prefix
./transcode.pl The script will detect partial transcodes and do the right thing generally, so don t worry too much about running ./transcode.pl too often. Results are being stored in various places including http://web.c9h.org/~cjac/perl/videos/

30 May 2020

Martin Michlmayr: ledger2beancount 2.2 released

I released version 2.2 of ledger2beancount, a ledger to beancount converter. Here are the changes in 2.2: You can get ledger2beancount from GitHub. Thanks to GitHub user MarinBernard for reporting a bug with virtual postings!

6 April 2020

Martin Michlmayr: ledger2beancount 2.1 released

I released version 2.1 of ledger2beancount, a ledger to beancount converter. Here are the changes in 2.1: You can get ledger2beancount from GitHub. Thanks to Thierry (thdox) for reporting a bug and for fixing some typos in the documentation. Thanks to Stefano Zacchiroli for some good feedback.

26 January 2017

Martin Michlmayr: Debian on Jetson TX1

Debian is now working on the NVIDIA Jetson TX1 developer kit, a development board based on the Tegra X1 chip (Tegra 210), a 64-bit ARM chip. We have a pre-built u-boot image in Debian as well as kernel and installer support. There are some minor kernel glitches but NVIDIA is very active upstream and I hope they'll get resolved soon. The Jetson TX1 developer kit makes a pretty good 64-bit ARM development platform. The board is supported in mainline u-boot and the mainline kernel and NVIDIA are pretty responsive to bug reports. Unfortunately, a proprietary blob is required for USB (and Ethernet is connected via USB). If you're interested in a good 64-bit ARM development platform, give Debian on the Jetson TX1 development kit a try.

7 January 2017

Enrico Zini: Teamwork

When I saw this video or this video I thought of this article. When I feel part of a tightly coordinated and synchronized team I feel proud for the achievements of the team as a whole, which I see as bigger than what I could have achieved alone. I also don't feel at risk of taking bad decisions. I feel less responsible. If I do what I'm told, I can't be blamed for doing the wrong things. I find it relaxing, every once in a while, to not have to be in charge. I guess this could be part of the allure of a totalitarian regime: being freed from the burden of growing up Thinking about this, reading those articles about romantic relationships, I see quite a bit of parallels also with organising cooperation and teamwork. It looks like I ended up making parallels between Polyamory, Anarchism, and Free Software again. If you think there should traditionally be also a mention of BDSM, go back to "I find it relaxing, every once in a while, to not have to be in charge".

25 July 2016

Martin Michlmayr: Debian on Jetson TK1

Debian on Jetson TK1 I became interested in running Debian on NVIDIA's Tegra platform recently. NVIDIA is doing a great job getting support for Tegra upstream (u-boot, kernel, X.org and other projects). As part of ensuring good Debian support for Tegra, I wanted to install Debian on a Jetson TK1, a development board from NVIDIA based on the Tegra K1 chip (Tegra 124), a 32-bit ARM chip. Ian Campbell enabled u-boot and Linux kernel support and added support in the installer for this device about a year ago. I updated some kernel options since there has been a lot of progress upstream in the meantime, performed a lot of tests and documented the installation process on the Debian wiki. Wookey made substantial improvements to the wiki as well. If you're interested in a good 32-bit ARM development platform, give Debian on the Jetson TK1 a try. There's also a 64-bit board. More on that later...

22 July 2016

Martin Michlmayr: Debian on Seagate Personal Cloud and Seagate NAS

The majority of NAS devices supported in Debian are based on Marvell's Kirkwood platform. This platform is quite dated now and can only run Debian's armel port. Debian now supports the Seagate Personal Cloud and Seagate NAS devices. They are based on Marvell's Armada 370, a platform which can run Debian's armhf port. Unfortunately, even the Armada 370 is a bit dated now, so I would not recommend these devices for new purchases. If you have one already, however, you now have the option to run native Debian. There are some features I like about the Seagate NAS devices: If you have a Seagate Personal Cloud and Seagate NAS, you can follow the instructions on the Debian wiki. If Seagate releases more NAS devices on Marvell's Armada platform, I intend to add Debian support.

13 March 2016

Antoine Beaupr : State of Mapping on the Debian Desktop

TL;DR: Mapbox Studio classic is not as good as Tilemill, Mapbox studio is very promising, but you still need to signup for access, and kosmtik seems to be working right now. Use kosmtik and help getting it in Debian.

The requirement I have been following the development of fascinating tools from the Development seed people, now seemingly all focused on the Mapbox brand. They created what seemed to me a revolutionary desktop tool called Tilemill. Tilemill allowed you to create custom map stylesheets on your desktop to outline specific areas or patterns or "things". My interest was to create outdoor maps - OSM has pretty good biking maps, but no generic outdoor tiles out there (I need stuff for skiing, canoeing, biking)... Mapbox has Mapbox outdoors but that's a paid plan as well. Oh and there's a place to download Garmin data files especially made for outdoors as well, and that could be loaded in Qmapshack, but they don't have coverage in Canada at all. Besides, I would like to print maps, I know, crazy... So I have been looking forward to seeing Tilemill packaged in Debian given how annoying it is to maintain Node apps (period). Unfortunately, by the time Debian people figured out all the Node dependencies, the Tilemill project stopped and has been stalled since 2012. It seems the Mapbox people have now been working on other products, and in the meantime, the community, scratching their heads, just switched to other projects.

Overview of alternatives I wrote this long post to the Debian bugtrackers to try to untangle all of this, but figured this would be interesting to a wider community than the narrow group of people working on Javascript packages, so I figured I would send this on Debian planet. So Here's a summary of what happened so far, after Tilemill development stopped. Hang on to your tiles boys and girls, there's a lot going on!

Mapbox Studio classic Mapbox people have released a new product in September 2014 named Mapbox studio classic. The code is still freely available and seems to be a fork of tilemill. Mapbox classic still has releases on github, last one is from November 2015. It looks like Mapbox studio classic has some sort of Mapbox.com lock-in, and there are certainly new copyright issues, if only with the bundled fonts, but it could probably be packaged after addressing those issues. There is an ITP for Mapbox Studio classic as well.

Mapbox Studio Then there's Mapbox Studio, which is a full rewrite of Mapbox Studio classic. You need to "signup" somehow to get access, even though parts of the code are free, namely the Mapbox GL studio project. It is an interesting project because it aims to make all this stuff happen in a web browser, which means it "should" work everywhere. Unfortunately for us, it means it doesn't work anywhere without a signup form, so that's out for me at least. There is an [ITP for Mapbox-studio][] yet it is unclear to me what that one means because the source code to Mapbox-studio doesn't seem to be available, as far as i can tell (and the ITP doesn't say either). That is actually the ITP for Mapbox studio classic.

Kosmtik The Openstreetmap-carto developers have mostly switched to kosmtik instead of Mapbox. Kosmtik is another Node desktop app that seems fairly lightweight and mostly based on plugins. Ross has an ITP for kosmtik. The package is waiting on other node dependencies to be uploaded (yes, again).

The future of Tilemill And there's still this RFP for tilemill, which should probably be closed now because the project seems dead and plenty of alternatives exist. I wonder if node some dependencies that were packaged for Tilemill actually now need to be removed from Debian, because they have become useless leaf packages... I am leaving the Tilemill RFP open for someone to clean that up.

CartoCSS Oh, and finally one could mention another Mapbox project, Carto, a command line CSS tools that implements some sort of standard CSS language that all those tools end up using to talk to Mapnik, more or less. There are no RFPs for that.

24 January 2016

Martin Michlmayr: QNAP TS-x09 installer available again

Debian 8.3 came out today. As part of this update, Debian installer images for QNAP TS-109, TS-209 and TS-409 are available again. These devices are pretty old but there are still some users. We dropped installer support several years ago because the installer ramdisk was too large to fit in flash. Since then, users had to install Debian 6.0 (squeeze) and upgrade from there. When squeeze was removed from the Debian mirrors recently, I received mail from a number of users. I investigated a bit and found out that we can bring back the installer thanks to XZ compression and some other changes. The installer is available for jessie and stretch.

7 October 2015

Steve Kemp: Generating fingerprints from SSH keys

I've been allowing users to upload SSH public-keys, and displaying them online in a form. Displaying an SSH public key is a pain, because they're typically long. That means you need to wrap them, or truncate them, or you introduce a horizontal scroll-bar. So rather than displaying them I figured I'd generate a fingerprint when the key was uploaded and show that instead - This is exactly how github shows your ssh-keys. Annoyingly there is only one reasonable way to get a fingerprint from a key: You can sometimes calculate the key via more direct, but less obvious methods:
awk ' print $2 ' ~/.ssh/id_rsa.pub   base64 -d   md5sum
But that won't work for all key-types. It is interesting to look at the various key-types which are available these days:
mkdir ~/ssh/
cd ~/ssh/
for i in dsa ecdsa ed25519 rsa rsa1 ; do
  ssh-keygen -P "" -t $i -f $ i -key
done
I've never seen an ed25519 key in the wild. It looks like this:
$ cat ~/ssh/ed25519-key.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMcT04t6UpewqQHWI4gfyBpP/ueSjbcGEze22vdlq0mW skx@shelob
Similarly curve-based keys are short too, but not as short:
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLTJ5+  \
 rWoq5cNcjXdhzRiEK3Yq6tFSYr4DBsqkRI0ZqJdb+7RxbhJYUOq5jsBlHUzktYhOahEDlc9Lezz3ZUqXg= skx@shelob
Remember what I said about wrapping? Ha! Anyway for the moment I've hacked up a simple perl module SSH::Key::Fingerprint which will accept a public key and return the fingerprint, as well as validating the key is well-formed and of a known-type. I might make it public in the future, but I think the name is all wrong. The only code I could easily find to do a similar job is this node.js package, but it doesn't work on all key-types. Shame. And that concludes this weeks super-happy fun-time TODO-list item.

24 July 2015

Martin Michlmayr: Congratulations to Stefano Zacchiroli

Stefano Zacchiroli receiving the O'Reilly Open Source Award I attended OSCON's closing sessions today and was delighted to see my friend Stefano Zacchiroli (Zack) receive an O'Reilly Open Source Award. Zack acted as Debian Project Leader for three years, is working on important activities at the Open Source Initiative and the Free Software Foundation, and is generally an amazing advocate for free software. Thanks for all your contributions, Zack, and congratulations!

Next.