I recently bought a wireless / phone adapter / VDSL modem from my Internet Service Provider (ISP) during my last outage. It generally works fine as a VDSL modem, but unfortunately, I can't seem to get used to configuring the device through their clickety web user interface... Furthermore, I am worried that I can't backup the config in a meaningful way, that is: if the device fails, I will probably not find the same model again and because they run a custom Linux distributions, the chances of the backup being possible to restore on another machine are basically zero. No way i will waste my time configuring this black box. So I started looking at running a distribution like OpenWRT on it.

(Unfortunately, I don't even dare hoping to run a decent operating system like Debian on those devices, if only because of the exotic chipsets that require all sorts of nasty hacks to run...)

The machine is a SmartRG SR630n (specs). I am linking to third party site, because the SmartRG site doesn't seem to know about their own product (!). I paid extra for this device to get one that would do both Wifi and VoIP, so i could replace two machines: my current Soekris net5501 router and a Cisco ATA 186 phone adapter that seems to mysteriously defy the challenges of time. (I don't remember when I got that thing, but it's at least from 2006.) Unfortunately, it seems that SmartRG are running a custom, proprietary Linux distribution. According to my ISP, init is a complete rewrite that reads an XML config file (and indeed it's the format of the backup files) and does the configuration through a shared memory scheme (!?). According to DSL reports, the device seems to be running a Broadcom 63168 SOC (system on a chip) that is unsupported in Linux. There are some efforts to write drivers for those from scratch, but they have been basically stalled for years now. Here are more details on the sucker:

• walkthrough - i can easily get telnet access and a busybox shell
• ls -lR - all files listing
• adsl information - all sorts of cryptic information about the DSL line
• dumpsysinfo output - some custom information dump tool, includes dmesg, kernel modules listing and so on
• xml config dump - some passwords were obviously censored...
• image dump - extracted straight from /dev/mtdblock0 with nc
• extracted image dump - used a patched version of squashfs-tools to support special LZMA magic markers
• interesting binary blobs - a select few files that seem to hold firmware images or interesting proprietary blobs

Now the next step would logically be to "simply" build a new image with OpenWRT and install it in place. Then I would need to figure out a way to load the binary blobs into the OpenWRT kernel and run all the ADSL utilities as well. It's basically impossible: the odds of the binary modules being compatible with another arbitrary release of the Linux kernel are near zero. Furthermore, the userland tool are most likely custom as well. And worse of all: it seems that Bell Canada deployed a custom "Lucent Stinger" DSLAM which requires a custom binary firmware in the modem. This could be why the SmartRG is so bizarre in the first place. As long as the other end is non-standard, we are all screwed. And those Stinger DSLAM will stick around for a long time, thanks to bell.

See this other good explanation of Stinger.

Which means this machine is now yet another closed box sitting on the internet without firmware upgrades, totally handicapped. I will probably end up selling it back for another machine that has OpenWRT support for their VDSL modems. But there are very few such machines, and with a lot of those, VDSL support is often marked as "spotty" or "in progress". Some machines are supported but are basically impossible to find. There's the Draytek modems are also interesting because, apparently, some models run OpenWRT out of the box too, which is a huge benefit. This is because they use the more open Lantiq SOC. Which are probably not going to support Stinger lines. Still, there are some very interesting projects out there... The Omnia is one I am definitely interested in right now. I really like their approach... But then they don't have a VDSL chipset in there (I asked for one, actually). And the connectors are only mini-PCIe, which makes it impossible to connect a VDSL PCI card into it. I could find a single VDSL2 PCI card online, and it could be supported, but only the annex B is available, not the annex A, and it seems the network is using "annex A" according to the ADSL stats i had in 2015-05-28-anarcat-back-again. With such a card, I could use my existing Soekris net5501 router, slam a DSL card into it, and just use the SmartRG as a dumb wifi router/phone adapter. Then it will remain to see how supported are those VDSL cards in FreeBSD (they provide Linux source code, so that's cool). And of course, all this assumes the card works with the "Stinger" mode, which is probably not the case anyways. Besides, I have VDSL2 here, not the lowly ADSL2+. By the way, Soekris keeps on pushing new interesting products out: their net6501, with 2 extra Gig-E cards could be a really interesting high-end switch, all working with free software tools. A friend has a SmartRG 505n modem, which looks quite similar, except without the ATA connectors. And those modems are the ones that Teksavvy recommends ("You may use a Cellpipe 7130 or Sagemcom F@ST 2864 in lieu of our SmartRG SR505N for our DSL 15/10, DSL 25 or DSL 50 services."). Furthermore, Teksavvy provides a firmware update for the 505n - again, no idea if it works with the 630n. Of course, the 505n doesn't run OpenWRT either. So, long story short, again I got screwed by my ISP: I thought i would get a pretty hackable device, "running Linux" that my ISP said over the phone. I got weeks of downtime, no refund, and while i got a better line (more reliable, higher bandwidth), my costs doubled. And I have yet another computing device to worry about: instead of simplifying and reducing waste, I actually just added crap on top of my already cluttered desk. Next time, maybe I'll tell you about how my ISP overbilled me, broke IPv6 and drops large packets to the floor. I haven't had a response from them in months now... hopefully they will either answer and fix all of this (doubtful) or I'll switch to some other provider, probably Teksavvy. Many thanks to the numerous people in the DSL reports Teksavvy forum that have amazing expertise. They are even building a map of Bell COs... Thanks also to Taggart for helping me figure out how the firmware images work and encouraging me to figure out how my machine works overall.

Note: all the information shared here is presented in the spirit of the fair use conditions of copyright law.

So the other day, I was able to do this in an Ubuntu natty amd64 chroot for the first time.

# cat > /etc/apt/apt.conf.d/multiarch-me
APT::Architectures   "amd64"; "i386";  ;
^D
# cat >> /etc/dpkg/dpkg.cfg
foreign-architecture i386
^D
# apt-get update
# apt-get install flashplugin-installer:i386
Reading package lists... Done
Building dependency tree       
Reading state information... Done
[...]
The following NEW packages will be installed:
  flashplugin-installer:i386 flashplugin-nonfree:i386 gcc-4.5-base:i386
  libatk1.0-0:i386 libavahi-client3:i386 libavahi-common-data:i386
  libavahi-common3:i386 libc6:i386 libcairo2:i386 libcomerr2:i386
  libcups2:i386 libdatrie1:i386 libdbus-1-3:i386 libdrm2:i386
  libegl1-mesa:i386 libexpat1:i386 libfontconfig1:i386 libfreetype6:i386
  libgcc1:i386 libgcrypt11:i386 libgdk-pixbuf2.0-0:i386 libgl1-mesa-glx:i386
  libglib2.0-0:i386 libgnutls26:i386 libgpg-error0:i386 libgssapi-krb5-2:i386
  libgtk2.0-0:i386 libgtk2.0-common libice6:i386 libjasper1:i386
  libjpeg62:i386 libk5crypto3:i386 libkeyutils1:i386 libkrb5-3:i386
  libkrb5support0:i386 libnspr4:i386 libnspr4-0d:i386 libnss3:i386
  libnss3-1d:i386 libpango1.0-0:i386 libpcre3:i386 libpixman-1-0:i386
  libpng12-0:i386 libselinux1:i386 libsm6:i386 libsqlite3-0:i386
  libstdc++6:i386 libtasn1-3:i386 libthai-data libthai0:i386 libtiff4:i386
  libudev0:i386 libuuid1:i386 libx11-6:i386 libx11-data libx11-xcb1:i386
  libxau6:i386 libxcb-dri2-0:i386 libxcb-render0:i386 libxcb-shape0:i386
  libxcb-shm0:i386 libxcb-xfixes0:i386 libxcb1:i386 libxcomposite1:i386
  libxcursor1:i386 libxdamage1:i386 libxdmcp6:i386 libxext6:i386
  libxfixes3:i386 libxft2:i386 libxi6:i386 libxinerama1:i386 libxrandr2:i386
  libxrender1:i386 libxt6:i386 libxxf86vm1:i386 x11-common zlib1g:i386
0 upgraded, 78 newly installed, 0 to remove and 3 not upgraded.
Need to get 15.1 MB/15.6 MB of archives.
After this operation, 48.9 MB of additional disk space will be used.
Do you want to continue [Y/n]?

It is a truly heady experience, after so many years of talking about the need to properly support multiarch in Debian and Ubuntu, to see support for cross-installation of packages come to fruition. If you've talked to me any time in the past couple of weeks and noticed it's a little hard to get me to change the subject, well, that's why. Many who have grown accustomed to Debian and Ubuntu's lack of support for installing i386 packages on amd64 (or vice versa) may wonder what the fuss is about. (Whereas others who are well versed in distributions such as Red Hat and SuSE may laugh and wonder what took us so long...) So maybe a few words of explanation are in order. If you've ever installed ia32-libs on an amd64 machine anywhere; if you've ever noticed a bug where ia32-libs didn't work right because of wrong system paths, or had to file a request for another library to be added to ia32-libs because it wasn't included in the set of libraries Debian decided to package up in this grotesque, all-in-one 32-bit compatibility bundle; if you've ever decided not to install a 64-bit OS on your perfectly 64-bit-capable hardware because of concern that you wouldn't be able to run $random32-bitonly_application; multiarch is for you. If you've gotten stuck maintaining a lib32foo "biarch" package in Debian due to popular demand, multiarch is definitely for you. :) If you've ever cross-compiled software for a different Debian architecture than the one you were running, multiarch is for you. If you've ever wanted to run binaries for a different architecture under emulation, and found it awkward to set up the library dependencies, multiarch is for you, too. Because although the .deb world may be a little late to the party, we're also naturally taking things much further than anyone's done with rpm. Multiarch won't just give you the ability to install 32-bit libs on 64-bit systems; it'll give you the ability to install libs for any known architecture on any system. And a whole lot of pain just falls out of the equation in the process. A cross-compiling environment looks the same as a native-compiling environment. An emulated system looks the same as a native system. We can start to seriously consider cross-grading systems from one architecture to another. And all this is happening now. The groundwork is there in Ubuntu natty. Wheezy will be the release that brings multiarch to Debian. When dpkg 1.16.0 is uploaded to unstable real soon now, the bootstrapping will begin. I am immensely grateful to everyone who's helped make multiarch a reality - to Tollef, Matt and others for seeding the vision; Aurelien, Matthias and Arthur for their work to ready the toolchain; David and Michael for the apt implementation; Guillem and Raphael for the dpkg implementation, and Linaro's support to help make this possible; and the many other developers who've helped to refine this design over the years in numerous other BoFs, sessions and mailing list threads. I'm excited to find out what the Debian community will do with multiarch now that it's upon us. Christian, maybe you should start a pool for how long it will take before all the libraries shipped in ia32-libs have been converted to multiarch and we can drop ia32-libs from the archive?

John Penrose said:

"[...] the credit union movement in this country is exactly the kind of excellent initiative that [...] fulfils many of the necessary and vital functions that she has mentioned and is aimed at those at the bottom of the income scale."

Source I've emailed him a thank-you and been told that ABCUL will explain to Fiona Mactaggart how they're working to increase coverage and capacity. I've noticed credit unions in the south-west expanding their areas, or merging in order to serve more members. I think we've also got a credit union rep on the board of Cooperatives-SW again. Yay. Now, back to coding.

Seriously. Now that we have been using the request tracker for quite some time, it’s even more obvious that the DSA team is not up to its task. Use login “guest” and password “readonly” if you want to check the RT tickets linked in this article. The facts

• 65 public tickets open (and 5 private tickets)
  • Want to help and seek input from DSA? No response.
  • Want to help and you provide patches and/or solutions? Many are still waiting.
  • Security updates? Who cares…
  • Backup of ftp-master (ries)? No reply (though ftp-master stuff is mirrored on merkel)…
  • New alpha machine? No movement despite the fact that we still have no DD-accessible porter machine.
• 68 tickets closed. Here are some unscientific and manual stats (I counted one each time that someone was involved for the work or for closing the ticket):
  • 27 for James Troup (elmo)
  • 26 for Phil Hands (fil)
  • 13 for me (buxy)
  • 3 for Martin ‘Joey’ Schulze (those I manually forwarded him)
  • 3 for Ryan Murray (neuro)
  • 3 for Matt Taggart (taggart)
  • 2 for Josip Rodin (he handles tickets concerning the mirrors until they have a dedicated queue in the RT)

Note that myself and Matt do not have the needed rights to fix most of the tickets, so we provided help on a best-effort basis. Otherwise we would have done more. The communication problem It’s a multi-level problem. Each of the members has some problems with one or more other members. Joey’s behavior has been part of the recurring problems mentioned: he doesn’t use the RT, doesn’t read the DSA email alias and doesn’t follow the DSA IRC channel but he still does stuff very regularly without reporting anything and obviously problems happen. Ryan and James tried to impose him a rule to document what he does, without success apparently. On the other side, as far as I know, Ryan and James also don’t impose themselves to document everything in a central changelog. Joey has refused to provide me an explanation for his behavior. He just reminded me that he holds grudges against James and Ryan because as ftpmasters they didn’t cooperate well with him while he was stable release manager. In general, outside of all personal griefs that they might have, the DSA members do not communicate very much (at least not on their own official channels). Some examples have already been given concerning the request tracker, but it’s not much more effective on IRC. Most of the traffic on the channel is made up by local admins fixing the problems themselves without any intervention by any DSA. I also use the channel to regularly ping some DSA about simple issues and/or stuff that they usually handle. It used to work somewhat but lately fil has been busy (with the kernel summit and other conferences) and I simply got no answer at all… for example I pinged elmo, neuro and fil several times in the last weeks in the hope that they handle the tickets of the security team (#150, #157, #164) without results. There’s room for improvement. The leadership problem The team has no designated leader and every time that there’s a decision to take, they are blocked. Joey wouldn’t communicate and give his opinion, Ryan is extremely requiring and perfectionist, there’s not much room for compromise… A long time ago in a galaxy far, far away, Joey and elmo were friends. It’s even Joey who gave root rights to elmo. Nowadays, it’s rather James that is sort-of leading the team but he’s fed up of the situation and hasn’t managed to get out of this mess. He refuses to take drastic measures by himself because he’s not clearly the leader and doesn’t solicit a decision of the Debian leader (or the project) because he believes that the DSA team is not under the scope of the constitution! This can’t last any further. We’ll have to do something about it. Stay tuned.

It’s very difficult to discuss with DSA and make things evolve if none of the DSA member express an interest in something related to your goal: here comes an example of a story like another in my desperate quest to try to help the DSA team. Last time gluck ran out of space, a few non-DSA people (me, taggart, Ganneff, and others I might have forgotten) contacted people to ask them to clean their home directories. Following that discussion we discussed a bit about the opportunity to move some services from gluck on another host. Among the services on gluck, there’s cvs.debian.org. As an Alioth administrator, it struck me that cvs.debian.org is the only VCS service that’s handled by the DSA team. It seems logical to not duplicate the administrative work and have all the VCS repositories handled by the same team. The logical conclusion is that cvs.debian.org should be deprecated in favor of Alioth. So I made the suggestion in RT ticket #146 (login with guest/readonly). I got exactly zero response from DSA. No support and no opposition. So I went ahead and contacted the last users of cvs.debian.org:

• webwml: the website team
• debian-doc: the Debian documentation project
• debian-admin: the DSA team (this was already suggested in April this year in ticket #44, no response of course… except elmo saying me that he’s in favor. On IRC I also discovered that neuro doesn’t like bzr and is thus not in favor of such a move. Furthermore he visibly wants to keep control on userdir-ldap, thus he probably has not much interest in moving to a distributed system.)
• buildd: the Packages-arch-specific file is maintained in the dak cvs…

All in all, the debian-doc and debian-www folks are rather supportive of the move, but it requires adjustment to the build infrastructure, in particular to keep track of the status of translations. I have no answer from DSA and the buildd guys however. The web team started a wiki page to evaluate the VCS that they would switch to. Volunteers would be welcome to organize the conversion of the repositories and to fix the build infrastructure accordingly. This a nice little project for new contributors that want to learn.

Since march the DSA team uses a request tracker to handle all the requests that they get. Therefore you should no more use the debian-admin@debian.org alias (DSA members only). The debian-admin@lists.debian.org mailing list (DSA + local admins) is also scheduled for removal. Instead you should mail admin@rt.debian.org making sure to put “Debian RT” somewhere in the subject (crude but working spam prevention). By default, your ticket ends up in a private incoming queue. Up to now only DSA members had access to those queues. They moved most tickets in the public queue but it happened several times that this simple administrative burden took several weeks. Since a few days, Matt Taggart and myself have been granted RT accounts with required privileges to handle tickets and move them between the various queues. So from now on, all tickets will be quickly moved in the public queue (when they don’t contain sensitive information of course). We’ll be able to do some triage and ping DSA members for urgent requests. Matt and me have been trying to help out the DSA team for quite some time now, and while we’re not DSA with root rights, we’re in regular contact with the members (except Martin “Joey” Schulze who refuses to join the DSA IRC channel, who doesn’t use the request tracker and who doesn’t read the DSA mails either) and with several local admins. So if you have some troubles interacting with the DSA team, you can try to get in touch with us and we’ll see if we can help you. In too many cases people contact directly individual DSA members (Joey in particular). Please refrain from doing so, other volunteers (who might help you) won’t notice your request. Furthermore systematic request tracker usage helps us identifying what is well handled and what’s not. And it’s not more complicated than a simple email for you. Update: currently local admins are not subscribed to the request tracker. The RT setup will need some further adjustment for that, so the debian-admin@lists.debian.org list can still be useful if you want to reach them.

I'm currently sitting in Denver International awaiting a flight to Houston where I'll meet up with Taggart for the flight into Mexico. There we'll meet up with some other folks (Ryan Murray, Troy Heber & Junichi Uekawa at least) and take a cab down to Oaxtepec. We've had *tremendous* trouble booking our hotel room (and by we, I mean Troy). In fact, last we heard they were cancelling our reservation. I strongly suspect this is an attempt at bribe soliciation, or maybe they actually just don't want to do business with us? Not quite sure - my employer is sponsoring my travel here, and I haven't noticed a "bribe" category on the expense report.. maybe I can list it under tip? :) Oh well, I suspect the worst case is that we can find asylum for a night on someone's floor until our accomodations get worked out. I'm very much looking forward to arriving at DebConf & seeing everyone, not to mention trying the local cuisine :)