Search Results: "will"

15 January 2016

Ian Wienand: Australia, ipv6 and dd-wrt

It seems that other than Internode, no Australian ISP has any details at all about native IPv6 deployment. Locally I am on Optus HFC, which I believe has been sold to the NBN, who I believe have since discovered that it is not quite what they thought it was. i.e. I think they have more problems than rolling out IPv6 and I won't hold my breath. So the only other option is to use a tunnel of some sort, and it seems there is really only one option with local presence via SixXS. There are other options, notably He.net, but they do not have Australian tunnel-servers. SixXS is the only one I could find with a tunnel in Sydney. So first sign up for an account there. The process was rather painless and my tunnel was provided quickly. After getting this, I got dd-wrt configured and working on my Netgear WNDR3700 V4. Here's my terse guide, cobbled together from other bits and pieces I found. I'm presuming you have a recent dd-wrt build that includes the aiccu tool to create the tunnel, and are pretty familiar with logging into it, etc. Firstly, on dd-wrt make sure you have JFFS2 turned on for somewhere to install scripts. Go Administration, JFFS2 Support, Internal Flash Storage, Enabled. Next, add the aiccu config file to /jffs/etc/aiccu.conf
# AICCU Configuration
# Login information
username USERNAME
password PASSWORD
# Protocol and server listed on your tunnel
protocol tic
server tic.sixxs.net
# Interface names to use
ipv6_interface sixxs
# The tunnel_id to use
# (only required when there are multiple tunnels in the list)
#tunnel_id <your tunnel id>
# Be verbose?
verbose false
# Daemonize?
daemonize true
# Require TLS?
requiretls true
# Set default route?
defaultroute true
Now you can add a script to bring up the tunnel and interface to /jffs/config/sixxs.ipup (make sure you make it executable) where you replace your tunnel address in the ip commands.
# wait until time is synced
while [  date +%Y  -eq 1970 ]; do
sleep 5
done
# check if aiccu is already running
if [ -n " ps grep etc/aiccu grep -v grep " ]; then
aiccu stop
sleep 1
killall aiccu
fi
# start aiccu
sleep 3
aiccu start /jffs/etc/aiccu.conf
sleep 3
ip -6 addr add 2001:....:....:....::/64 dev br0
ip -6 route add 2001:....:....:....::/64 dev br0
sleep 5
#### BEGIN FIREWALL RULES ####
WAN_IF=sixxs
LAN_IF=br0
#flush tables
ip6tables -F
#define policy
ip6tables -P INPUT DROP
ip6tables -P FORWARD DROP
ip6tables -P OUTPUT ACCEPT
# Input to the router
# Allow all loopback traffic
ip6tables -A INPUT -i lo -j ACCEPT
#Allow unrestricted access on internal network
ip6tables -A INPUT -i $LAN_IF -j ACCEPT
#Allow traffic related to outgoing connections
ip6tables -A INPUT -i $WAN_IF -m state --state RELATED,ESTABLISHED -j ACCEPT
# for multicast ping replies from link-local addresses (these don't have an
# associated connection and would otherwise be marked INVALID)
ip6tables -A INPUT -p icmpv6 --icmpv6-type echo-reply -s fe80::/10 -j ACCEPT
# Allow some useful ICMPv6 messages
ip6tables -A INPUT -p icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
ip6tables -A INPUT -p icmpv6 --icmpv6-type packet-too-big -j ACCEPT
ip6tables -A INPUT -p icmpv6 --icmpv6-type time-exceeded -j ACCEPT
ip6tables -A INPUT -p icmpv6 --icmpv6-type parameter-problem -j ACCEPT
ip6tables -A INPUT -p icmpv6 --icmpv6-type echo-request -j ACCEPT
ip6tables -A INPUT -p icmpv6 --icmpv6-type echo-reply -j ACCEPT
# Forwarding through from the internal network
# Allow unrestricted access out from the internal network
ip6tables -A FORWARD -i $LAN_IF -j ACCEPT
# Allow some useful ICMPv6 messages
ip6tables -A FORWARD -p icmpv6 --icmpv6-type destination-unreachable -j ACCEPT
ip6tables -A FORWARD -p icmpv6 --icmpv6-type packet-too-big -j ACCEPT
ip6tables -A FORWARD -p icmpv6 --icmpv6-type time-exceeded -j ACCEPT
ip6tables -A FORWARD -p icmpv6 --icmpv6-type parameter-problem -j ACCEPT
ip6tables -A FORWARD -p icmpv6 --icmpv6-type echo-request -j ACCEPT
ip6tables -A FORWARD -p icmpv6 --icmpv6-type echo-reply -j ACCEPT
#Allow traffic related to outgoing connections
ip6tables -A FORWARD -i $WAN_IF -m state --state RELATED,ESTABLISHED -j ACCEPT
Now you can reboot, or run the script, and it should bring the tunnel up and you should be correclty firewalled such that packets get out, but nobody can get in. Back to the web-interface, you can now enable IPv6 with Setup, IPV6, Enable. You leave "IPv6 Type" as Native IPv6 from ISP. Then I enabled Radvd and added a custom config in the text-box to get DNS working with google DNS on hosts with:
interface br0
 
AdvSendAdvert on;
prefix 2001:....:....:....::/64
  
  ;
 RDNSS 2001:4860:4860::8888 2001:4860:4860::8844
  
  ;
 ;
(again, replace the prefix with your own) That is pretty much it; at this point, you should have an IPv6 network and it's most likely that all your network devices will "just work" with it. I got full scores on the IPv6 test sites on a range of devices. Unfortunately, even a geographically close tunnel still really kills latency; compare these two traceroutes:
$ mtr -r -c 1 google.com
Start: Fri Jan 15 14:51:18 2016
HOST: jj                          Loss%   Snt   Last   Avg  Best  Wrst StDev
1.  -- 2001:....:....:....::      0.0%     1    1.4   1.4   1.4   1.4   0.0
2.  -- gw-163.syd-01.au.sixxs.ne  0.0%     1   12.0  12.0  12.0  12.0   0.0
3.  -- ausyd01.sixxs.net          0.0%     1   13.5  13.5  13.5  13.5   0.0
4.  -- sixxs.sydn01.occaid.net    0.0%     1   13.7  13.7  13.7  13.7   0.0
5.  -- 15169.syd.equinix.com      0.0%     1   11.5  11.5  11.5  11.5   0.0
6.  -- 2001:4860::1:0:8613        0.0%     1   14.1  14.1  14.1  14.1   0.0
7.  -- 2001:4860::8:0:79a0        0.0%     1  115.1 115.1 115.1 115.1   0.0
8.  -- 2001:4860::8:0:8877        0.0%     1  183.6 183.6 183.6 183.6   0.0
9.  -- 2001:4860::1:0:66d6        0.0%     1  196.6 196.6 196.6 196.6   0.0
10. -- 2001:4860:0:1::72d         0.0%     1  189.7 189.7 189.7 189.7   0.0
11. -- kul01s07-in-x09.1e100.net  0.0%     1  194.9 194.9 194.9 194.9   0.0
$ mtr -4 -r -c 1 google.com
Start: Fri Jan 15 14:51:46 2016
HOST: jj                          Loss%   Snt   Last   Avg  Best  Wrst StDev
1. -- gateway                    0.0%     1    1.3   1.3   1.3   1.3   0.0
2. -- 10.50.0.1                  0.0%     1   11.0  11.0  11.0  11.0   0.0
3. -- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
4. -- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
5. -- ???                       100.0     1    0.0   0.0   0.0   0.0   0.0
6. -- riv4-ge4-1.gw.optusnet.co  0.0%     1   12.1  12.1  12.1  12.1   0.0
7. -- 198.142.187.20             0.0%     1   10.4  10.4  10.4  10.4   0.0
When you watch what is actually using ipv6 (the ipvfoo plugin for Chrome is pretty cool, it shows you what requests are going where), it's mostly all just traffic to really big sites (Google/Google Analytics, Facebook, Youtube, etc) who have figured out IPv6. Since these are exactly the type of places that have made efforts to get caching as close as possible to you (Google's mirror servers are within Optus' network, for example) and so you're really shooting yourself in the foot going around it using an external tunnel. The other thing is that I'm often hitting IPv6 mirrors and downloading larger things for work stuff (distro updates, git clones, image downloads, etc) which is slower and wasting someone else's bandwith for really no benefit. So while it's pretty cool to have an IPv6 address (and a fun experiment) I think I'm going to turn it off. One positive was that after running with it for about a month, nothing has broken -- which suggests that most consumer level gear in a typical house (phones, laptops, TVs, smart-watches, etc) is either ready or ignores it gracefully. Bring on native IPv6!

Norbert Preining: CJK fonts and Ghostscript: cjk-gs-integrate release 20160115.0

I have made a new release of cjk-gs-integrate, the script that searches for all kind of CJK fonts, and makes them available to Ghostscript (and if asked for also to TeX by linking). The update has been uploaded to CTAN (link) and it will land in TeX Live within the next days. cjk-ghostscript Changes made in this version are mostly related to link handling and removal: For more explanations concerning how to run cjk-gs-integrate, please see the dedicated page: CJK fonts and Ghostscript integration. For feedback and bug reports, please use the github project page. Enjoy.

14 January 2016

Petter Reinholdtsen: Always download Debian packages using Tor - the simple recipe

During his DebConf15 keynote, Jacob Appelbaum observed that those listening on the Internet lines would have good reason to believe a computer have a given security hole if it download a security fix from a Debian mirror. This is a good reason to always use encrypted connections to the Debian mirror, to make sure those listening do not know which IP address to attack. In August, Richard Hartmann observed that encryption was not enough, when it was possible to interfere download size to security patches or the fact that download took place shortly after a security fix was released, and proposed to always use Tor to download packages from the Debian mirror. He was not the first to propose this, as the apt-transport-tor package by Tim Retout already existed to make it easy to convince apt to use Tor, but I was not aware of that package when I read the blog post from Richard. Richard discussed the idea with Peter Palfrader, one of the Debian sysadmins, and he set up a Tor hidden service on one of the central Debian mirrors using the address vwakviie2ienjx6t.onion, thus making it possible to download packages directly between two tor nodes, making sure the network traffic always were encrypted. Here is a short recipe for enabling this on your machine, by installing apt-transport-tor and replacing http and https urls with tor+http and tor+https, and using the hidden service instead of the official Debian mirror site. I recommend installing etckeeper before you start to have a history of the changes done in /etc/.
apt install apt-transport-tor
sed -i 's% http://ftp.debian.org/%tor+http://vwakviie2ienjx6t.onion/%' /etc/apt/sources.list
sed -i 's% http% tor+http%' /etc/apt/sources.list
If you have more sources listed in /etc/apt/sources.list.d/, run the sed commands for these too. The sed command is assuming your are using the ftp.debian.org Debian mirror. Adjust the command (or just edit the file manually) to match your mirror. This work in Debian Jessie and later. Note that tools like apt-file only recently started using the apt transport system, and do not work with these tor+http URLs. For apt-file you need the version currently in experimental, which need a recent apt version currently only in unstable. So if you need a working apt-file, this is not for you. Another advantage from this change is that your machine will start using Tor regularly and at fairly random intervals (every time you update the package lists or upgrade or install a new package), thus masking other Tor traffic done from the same machine. Using Tor will become normal for the machine in question. On Freedombox, APT is set up by default to use apt-transport-tor when Tor is enabled. It would be great if it was the default on any Debian system.

Lunar: Reproducible builds: week 37 in Stretch cycle

What happened in the reproducible builds effort between January 3rd and January 9th 2016:

Toolchain fixes David Bremner uploaded dh-elpa/0.0.18 which adds a --fix-autoload-date option (on by default) to take autoload dates from changelog. Lunar updated and sent the patch adding the generation of .buildinfo to dpkg.

Packages fixed The following packages have become reproducible due to changes in their build dependencies: aggressive-indent-mode, circe, company-mode, db4o, dh-elpa, editorconfig-emacs, expand-region-el, f-el, geiser, hyena, js2-mode, markdown-mode, mono-fuse, mysql-connector-net, openbve, regina-normal, sml-mode, vala-mode-el. The following packages became reproducible after getting fixed: Some uploads fixed some reproducibility issues, but not all of them: Patches submitted which have not made their way to the archive yet:
  • #809780 on flask-restful by Chris Lamb: implement support for SOURCE_DATE_EPOCH in the build system.
  • #810259 on avfs by Chris Lamb: implement support for SOURCE_DATE_EPOCH in the build system.
  • #810509 on apt by Mattia Rizzolo: ensure a stable file order is given to the linker.

reproducible.debian.net Add 2 more armhf build nodes provided by Vagrant Cascadian. This added 7 more armhf builder jobs. We now run around 900 tests of armhf packages each day. (h01ger) The footer of each page now indicates by which Jenkins jobs build it. (h01ger)

diffoscope development diffoscope 45 has been released on January 4th. It features huge memory improvements when comparing large files, several fixes of squashfs related issues that prevented comparing two Tails images, and improve the file list of tar and cpio archive to be more precise and consistent over time. It also fixes a typo that prevented the Mach-O to work (Rainer M ller), improves comparisons of ELF files when specified on the command line, and solves a few more encoding issues.

Package reviews 134 reviews have been removed, 30 added and 37 updated in the previous week. 20 new fail to build from source issues were reported by Chris Lamb and Chris West. prebuilder will now skip installing diffoscope to save time if the build results are identical. (Reiner Herrmann)

Jan Wagner: Running Ghost blogging platform via Docker

When I was thinking about using Ghost, I did read the installations guide and then I just closed the browser window.
I didn't wanted to install npm, yet another package manager, and just hack init scripts. Not speaking about updating Ghost itself. Some weeks later I did think about using Ghost again. It has a nice Markdown Editor and some nice other features. Since everybody is jumping on the Docker band wagon actually and I had used it for some tests already, I thought trying the Ghost Docker image might be a good idea. If you are interested into how I did that, read on. I suppose you have installed a stock Debian Jessie.

Installing Docker

Pulling the Docker image Just in case you didn't, you need to (re)start docker to work with service docker restart
# docker pull ghost

Making Ghost (container image) run forever I did not like systemd in the first place for many reasons. But in some circumstances it makes sense. In case of handling a Docker container, using a systemd unit file makes life much easier.
# mkdir -p /srv/docker/ghost/
# cat > /etc/systemd/system/ghost.service << EOF
[Unit]
Description=GHost Service  
After=docker.service  
Requires=docker.service
[Service]
ExecStartPre=-/usr/bin/docker kill ghost  
ExecStartPre=-/usr/bin/docker rm ghost  
ExecStartPre=-/usr/bin/docker pull ghost  
ExecStart=/usr/bin/docker run  --name ghost --publish 2368:2368 --env 'NODE_ENV=production' --volume /srv/docker/ghost/:/var/lib/ghost ghost  
ExecStop=/usr/bin/docker stop ghost
[Install]
WantedBy=multi-user.target  
EOF  
# systemctl enable ghost && systemctl daemon-reload && systemctl start ghost 
This will start your container on start and even is looking for a new Docker image and is fetching it, if needed. If you don't like this behavior, just comment out the line in the config and reread it with systemctl daemon-reload. Now you should have listening something on port 2368:
# netstat -tapn   grep 2368
tcp6       0      0 :::2368                 :::*                    LISTEN      7061/docker-proxy  
Update: Jo l Dinel did send me a mail, that starting your Docker container with --restart always will take care that it is brought up again if Docker or (even) the whole system will get restarted. For real I used that before and might be a lightweight solution, but I liked the systemd unit file solution a lot more.

Persistent Data Thanks to the Docker mount option you can find all your data in /srv/docker/ghost/. So your blog will still have content, even if the ghost Docker images is updated:
# ls /srv/docker/ghost/
apps  config.js  data  images  themes  

Accessing the container To kick your ghost into production, it might be useful if you make it available on port 80 at least. This can be done for example by changing your Docker publish configuration or adding a DNAT to your firewall. But I would recommand using a proxy in front of your Docker container. This might be part of one of my next articles.

Rapha&#235;l Hertzog: Working as a paid LTS contributor

A Debian LTS logoWhile the details about how to join the set of paid contributors have always been public (here) we did not advertise this fact very much outside of the people already interested in LTS (and thus subscribed to debian-lts@lists.debian.org). But right now we would like to have a few more paid contributors on board and I m thus posting this call for volunteers. Who can apply? You need to meet those requirements: Even though Freexian is located in France and requires you to provide invoice in EUR, there are no conditions on your nationality or country of residence. For contributors outside of the Euro zone, Freexian is using Transferwise to pay them with minimal currency conversion costs (Paypal is also possible if nothing else works). The rate offered to paid contributors is the same for all (75 EUR/hour), it s based on a correct rate for independent contractors in western Europe. If the rate is very high for your own country, then be happy to be able to invoice Freexian at this rate and use this opportunity to work less (for money) and contribute more to Debian on your (now copious) free time. How does the work look like? If you apply, you will have to send us an SSH key so that you can have access to the internal git repository used for work. It contains a ledger file to track the hours funded by sponsors and how they have been dispatched to the various contributors. You can always know how many hours are assigned to you, how many can be invoiced, and so on. You will have to update it once a month to record the work you did (and indicate us where the report has been published). The repository also contains a README with many explanations about the workflow (how hours are dispatched, the delay you have to publish your report, etc) and a small helper script (./find-work) to match up the pending updates (registered in dla-needed.txt) with the popularity of the package among the sponsors. Now the work itself is relatively well documented in the LTS wiki. You will have to provide updates for packages that need an update. You have some freedom in selecting the packages but at some point you will have to work on packages that you don t know that are written in a language that you have almost not used. So you must be able to go out of your comfort zone and still do a good work. You must also be able to multi-task because in some cases you will get stuck on a particular update and you will have to seek help from the upstream developer (or from the Debian package maintainer). Don t expect to be able to do all your work hours in a single run thus don t wait until the last days of the month. Start early and dispatch your work hours over the month. From time to time, you will also have to handle the LTS frontdesk for one week. During this week, you need to spend a bit of time every day to triage the new CVE, to respond to questions on the mailing list, and to sponsor updates prepared by volunteers who do not have upload rights. Questions? Ask your questions in the comments and I will update this section with your questions and our answers. What if I have no prior experience with security updates? Start getting some experience. The LTS and security teams are open for anyone to join. Read their documentation and provide some updates that other contributors can sponsor. Before accepting you as paid contributor, we generally ask you to prepare one or two DLA on your free time just to make sure that you know the workflow and that you are up to the task. What if I have only X hours available for paid LTS work? In the git repository there s a file where you document how many work hours you can handle. You might get less than this amount, but we generally never assign less than 8 hours (to make sure that you can handle one complicated update from start to end, or your possible week of LTS frontdesk). You can adjust it each month or even opt-out if you are not available for whatever reason. But once you have been assigned work hours, it s important to actually do the work that you requested! How do I apply? Get in touch with me (as documented).

One comment Liked this article? Click here. My blog is Flattr-enabled.

13 January 2016

Norbert Preining: Ian Buruma: Wages of Guilt

Since moving to Japan, I got more and more interested in history, especially the recent history of the 20th century. The book I just finished, Ian Buruma (Wiki, home page) Wages of Guilt Memories of War in Germany and Japan (Independent, NYRB), has been a revelation for me. As an Austrian living in Japan, I am experiencing the discrepancy between these two countries with respect to their treatment of war legacy practically daily, and many of my blog entries revolve around the topic of Japanese non-reconciliation.
Willy Brandt went down on his knees in the Warsaw ghetto, after a functioning democracy had been established in the Federal Republic of Germany, not before. But Japan, shielded from the evil world, has grown into an Oskar Matzerath: opportunistic, stunted, and haunted by demons, which it tries to ignore by burying them in the sand, like Oskar s drum.
Ian Buruma, Wages of Guilt, Clearing Up the Ruins
Buruma-Wages_of_Guilt The comparison of Germany and Japan with respect to their recent history as laid out in Buruma s book throws a spotlight on various aspects of the psychology of German and Japanese population, while at the same time not falling into the easy trap of explaining everything with difference in the guilt culture. A book of great depth and broad insights everyone having even the slightest interest in these topics should read.
This difference between (West) German and Japanese textbooks is not just a matter of detail; it shows a gap in perception.
Ian Buruma, Wages of Guilt, Romance of the Ruins
Only thinking about giving a halfway full account of this book is something impossible for me. The sheer amount of information, both on the German and Japanese side, is impressive. His incredible background (studies of Chinese literature and Japanese movie!) and long years as journalist, editor, etc, enriches the book with facets normally not available: In particular his knowledge of both the German and Japanese movie history, and the reflection of history in movies, were complete new aspects for me (see my recent post (in Japanese)). The book is comprised of four parts: The first with the chapters War Against the West and Romance of the Ruins; the second with the chapters Auschwitz, Hiroshima, and Nanking; the third with History on Trial, Textbook Resistance, and Memorials, Museums, and Monuments; and the last part with A Normal Country, Two Normal Towns, and Clearing Up the Ruins. Let us look at the chapters in turn: The boook somehow left me with a bleak impression of Japanese post-war times as well as Japanese future. Having read other books about the political ignorance in Japan (Norma Field s In the realm of a dying emperor, or the Chibana history), Buruma s characterization of Japanese politics is striking. He couldn t foresee the recent changes in legislation pushed through by the Abe government actually breaking the constitution, or the rewriting of history currently going on with respect to comfort women and Nanking. But reading his statement about Article Nine of the constitution and looking at the changes in political attitude, I am scared about where Japan is heading to:
The Nanking Massacre, for leftists and many liberals too, is the main symbol of Japanese militarism, supported by the imperial (and imperialist) cult. Which is why it is a keystone of postwar pacifism. Article Nine of the constitution is necessary to avoid another Nanking Massacre. The nationalist right takes the opposite view. To restore the true identity of Japan, the emperor must be reinstated as a religious head of state, and Article Nine must be revised to make Japan a legitimate military power again. For this reason, the Nanking Massacre, or any other example of extreme Japanese aggression, has to be ignored, softened, or denied.
Ian Buruma, Wages of Guilt, Nanking
While there are signs of resistance in the streets of Japan (Okinawa and the Hanako bay, the demonstrations against secrecy law and reversion of the constitution), we are still to see a change influenced by the people in a country ruled and distributed by oligarchs. I don t think there will be another Nanking Massacre in the near future, but Buruma s books shows that we are heading back to a nationalistic regime similar to pre-war times, just covered with a democratic veil to distract critics.
I close with several other quotes from the book that caught my attention: In the preface and introduction:
[ ] mainstream conservatives made a deliberate attempt to distract people s attention from war and politics by concentrating on economic growth.
The curious thing was that much of what attracted Japanese to Germany before the war Prussian authoritarianism, romantic nationalism, pseudo-scientific racialism had lingered in Japan while becoming distinctly unfashionable in Germany.
In Romance of the Ruins:
The point of all this is that Ikeda s promise of riches was the final stage of what came to be known as the reverse course, the turn away from a leftist, pacifist, neutral Japan a Japan that would never again be involved in any wars, that would resist any form of imperialism, that had, in short, turned its back for good on its bloody past. The Double Your Incomes policy was a deliberate ploy to draw public attention away from constitutional issues.
In Hiroshima:
The citizens of Hiroshima were indeed victims, primarily of their own military rulers. But when a local group of peace activists petitioned the city of Hiroshima in 1987 to incorporate the history of Japanese aggression into the Peace Memorial Museum, the request was turned down. The petition for an Aggressors Corner was prompted by junior high school students from Osaka, who had embarrassed Peace Museum officials by asking for an explanation about Japanese responsibility for the war.
The history of the war, or indeed any history, is indeed not what the Hiroshima spirit is about. This is why Auschwitz is the only comparison that is officially condoned. Anything else is too controversial, too much part of the flow of history .
In Nanking, by the governmental pseudo-historian Tanaka:
Unlike in Europe or China, writes Tanaka, you won t find one instance of planned, systematic murder in the entire history of Japan. This is because the Japanese have a different sense of values from the Chinese or the Westerners.
In History on Trial:
In 1950, Becker wrote that few things have done more to hinder true historical self-knowledge in Germany than the war crimes trials. He stuck to this belief. Becker must be taken seriously, for he is not a right-wing apologist for the Nazi past, but an eminent liberal.
There never were any Japanese war crimes trials, nor is there a Japanese Ludwigsburg. This is partly because there was no exact equivalent of the Holocaust. Even though the behavior of Japanese troops was often barbarous, and the psychological consequences of State Shinto and emperor worship were frequently as hysterical as Nazism, Japanese atrocities were part of a military campaign, not a planned genocide of a people that included the country s own citizens. And besides, those aspects of the war that were most revolting and furthest removed from actual combat, such as the medical experiments on human guinea pigs (known as logs ) carried out by Unit 731 in Manchuria, were passed over during the Tokyo trial. The knowledge compiled by the doctors of Unit 731 of freezing experiments, injection of deadly diseases, vivisections, among other things was considered so valuable by the Americans in 1945 that the doctors responsible were allowed to go free in exchange for their data.
Some Japanese have suggested that they should have conducted their own war crimes trials. The historian Hata Ikuhiko thought the Japanese leaders should have been tried according to existing Japanese laws, either in military or in civil courts. The Japanese judges, he believed, might well have been more severe than the Allied tribunal in Tokyo. And the consequences would have been healthier. If found guilty, the spirits of the defendants would not have ended up being enshrined at Yasukuni. The Tokyo trial, he said, purified the crimes of the accused and turned them into martyrs. If they had been tried in domestic courts, there is a good chance the real criminals would have been flushed out.
After it was over, the Nippon Times pointed out the flaws of the trial, but added that the Japanese people must ponder over why it is that there has been such a discrepancy between what they thought and what the rest of the world accepted almost as common knowledge. This is at the root of the tragedy which Japan brought upon herself.
Emperor Hirohito was not Hitler; Hitler was no mere Shrine. But the lethal consequences of the emperor-worshipping system of irresponsibilities did emerge during the Tokyo trial. The savagery of Japanese troops was legitimized, if not driven, by an ideology that did not include a Final Solution but was as racialist as Hider s National Socialism. The Japanese were the Asian Herrenvolk, descended from the gods.
Emperor Hirohito, the shadowy figure who changed after the war from navy uniforms to gray suits, was not personally comparable to Hitler, but his psychological role was remarkably similar.
In fact, MacArthur behaved like a traditional Japanese strongman (and was admired for doing so by many Japanese), using the imperial symbol to enhance his own power. As a result, he hurt the chances of a working Japanese democracy and seriously distorted history. For to keep the emperor in place (he could at least have been made to resign), Hirohito s past had to be freed from any blemish; the symbol had to be, so to speak, cleansed from what had been done in its name.
In Memorials, Museums, and Monuments:
If one disregards, for a moment, the differences in style between Shinto and Christianity, the Yasukuni Shrine, with its relics, its sacred ground, its bronze paeans to noble sacrifice, is not so very different from many European memorials after World War I. By and large, World War II memorials in Europe and the United States (though not the Soviet Union) no longer glorify the sacrifice of the fallen soldier. The sacrificial cult and the romantic elevation of war to a higher spiritual plane no longer seemed appropriate after Auschwitz. The Christian knight, bearing the cross of king and country, was not resurrected. But in Japan, where the war was still truly a war (not a Holocaust), and the symbolism still redolent of religious exultation, such shrines as Yasukuni still carry the torch of nineteenth-century nationalism. Hence the image of the nation owing its restoration to the sacrifice of fallen soldiers.
In A Normal Country:
The mayor received a letter from a Shinto priest in which the priest pointed out that it was un-Japanese to demand any more moral responsibility from the emperor than he had already taken. Had the emperor not demonstrated his deep sorrow every year, on the anniversary of Japan s surrender? Besides, he wrote, it was wrong to have spoken about the emperor in such a manner, even as the entire nation was deeply worried about his health. Then he came to the main point: It is a common error among Christians and people with Western inclinations, including so-called intellectuals, to fail to grasp that Western societies and Japanese society are based on fundamentally different religious concepts . . . Forgetting this premise, they attempt to place a Western structure on a Japanese foundation. I think this kind of mistake explains the demand for the emperor to bear full responsibility.
In Two Normal Towns:
The bust of the man caught my attention, but not because it was in any way unusual; such busts of prominent local figures can be seen everywhere in Japan. This one, however, was particularly grandiose. Smiling across the yard, with a look of deep satisfaction over his many achievements, was Hatazawa Kyoichi. His various functions and titles were inscribed below his bust. He had been an important provincial bureaucrat, a pillar of the sumo wrestling establishment, a member of various Olympic committees, and the recipient of some of the highest honors in Japan. The song engraved on the smooth stone was composed in praise of his rich life. There was just one small gap in Hatazawa s life story as related on his monument: the years from 1941 to 1945 were missing. Yet he had not been idle then, for he was the man in charge of labor at the Hanaoka mines.
In Clearing Up the Ruins:
But the question in American minds was understandable: could one trust a nation whose official spokesmen still refused to admit that their country had been responsible for starting a war? In these Japanese evasions there was something of the petulant child, stamping its foot, shouting that it had done nothing wrong, because everybody did it.
Japan seems at times not so much a nation of twelve-year-olds, to repeat General MacArthur s phrase, as a nation of people longing to be twelve-year-olds, or even younger, to be at that golden age when everything was secure and responsibility and conformity were not yet required.
For General MacArthur was right: in 1945, the Japanese people were political children. Until then, they had been forced into a position of complete submission to a state run by authoritarian bureaucrats and military men, and to a religious cult whose high priest was also formally chief of the armed forces and supreme monarch of the empire.
I saw Jew S ss that same year, at a screening for students of the film academy in Berlin. This showing, too, was followed by a discussion. The students, mostly from western Germany, but some from the east, were in their early twenties. They were dressed in the international uniform of jeans, anoraks, and work shirts. The professor was a man in his forties, a 68er named Karsten Witte. He began the discussion by saying that he wanted the students to concentrate on the aesthetics of the film more than the story. To describe the propaganda, he said, would simply be banal: We all know the what, so let s talk about the how. I thought of my fellow students at the film school in Tokyo more than fifteen years before. How many of them knew the what of the Japanese war in Asia.

Keith Packard: auto-calendar

Automatic Calendar Management Notmuch + Calypso One of the big features of outlook/exchange in my world is automatically merging of incoming calendar updates from email. This makes my calendar actually useful in knowing what meetings people have asked me to attend. As I'm not willing to otherwise tolerate outlook, I decided to try and provide that in my preferred environment; notmuch and calypso. Identifying calendar updates The first trick is how to identify incoming messages with calendar updates. I'd love to be able to search for specific mime content types, but I haven't been able to construct such a search. Failing that, I'm just looking for messages containing the string 'text/calendar':
notmuch search --output=messages tag:inbox AND text/calendar
Next, I want to skip over previously scanned calendar updates, so I'll plan on tagging messages that have been scanned with the 'calendar' tag and skip those:
notmuch search --output=messages tag:inbox AND text/calendar AND not tag:calendar
jq sed for json With the messages containing likely calendar entries identified, the remaining task is to extract the MIME section containing the actual calendar data. Notmuch can generate json for the message, leaving us only needing to parse the json and extract the relevant section. I found the 'jq' tool in the archive, which looks like a rather complicated parsing and reformatting tool for json data. It took a while to understand, but I managed to generate a command string that takes a notmuch message and pulls out the content for all text/calendar elements:
jq -r '..  select(."content-type"? == "text/calendar")   .content'
This is a recursive walk over the data structure. It looks for structures with "content-type": "text/calendar" and dumps their "content" elements in raw text form. Putting it all together Here's the complete script:
#!/bin/bash
SEARCH="tag:inbox AND not tag:calendar AND text/calendar"
TMP= mktemp 
trap "rm -r $TMP" 0 1 15
notmuch search --output=messages $SEARCH   while read message; do
    notmuch show --format=json $message   
        jq -r '..   select(."content-type"? == "text/calendar")   .content' > $TMP
    if [ -s $TMP ]; then
        calypso --import private/calendar $TMP && notmuch tag +calendar $message
    else
        notmuch tag +calendar $message
    fi
done
I'd love to fix calypso's --import operation to talk to the local calypso daemon with the database loaded; the current mechanism actually loads the entire database into a new process and then applies the new data to that. With my calendar often containing hundreds of entries, that takes a while.

Rapha&#235;l Hertzog: Freexian s report about Debian Long Term Support, December 2015

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In December, 113.50 work hours have been dispatched among 9 paid contributors. Their reports are available: Evolution of the situation We lost our first silver sponsor (Gandi.net, they prefer to give the same amount of money to Debian directly) and another sponsor reduced his sponsorship level. While this won t show in the hours dispatched in January, we will do a small jump backwards in February (unless we get new sponsors replacing those in the next 3 weeks). This is a bit unfortunate as we are rather looking at reinforcing the amount of sponsorship we get as we approach Wheezy LTS and we will need more support to properly support virtualization related packages and other packages that were formerly excluded from Squeeze LTS. Can you convince your company and help us reach our second goal? In terms of security updates waiting to be handled, the situation is close to last month. It looks like that having about 20 packages needing an update is the normal situation and that we can t really get further down given the time required to process some updates (sometimes we wait until the upstream authors provides a patch, and so on). Thanks to our sponsors We got one new bronze sponsor but he s not listed (he did not fill the form where we request their permission to be listed).

2 comments Liked this article? Click here. My blog is Flattr-enabled.

Rhonda D'Vine: 2016 Resolutions

People these days often do think about what worked well in the last year that they are proud of, what didn't work so well and what they plan to change the coming year. For me a fair amount of the resolutions were about my name. One of them was getting rid of my old name from the Debian Project Participants page. Actually, I started with it on new year's eve already:
DatePackageVersion
Dec 31abook0.6.1-1
Jan 01tworld1.3.2-1
Jan 01blosxom2.1.2-2
Jan 02netris0.52-10
Jan 03t-prot3.4-4
Jan 04rungetty1.2-16
Jan 05tworld1.3.2-2
Jan 06tetrinet0.11+CVS20070911-2
Jan 07xblast-tnt-musics20050106-3
Jan 08xblast-tnt-sounds20040429-3
Jan 09xblast-tnt-levels20050106-3
Jan 10xblast-tnt-images20050106-3
Jan 11tetradraw2.0.3-9
Jan 12ldapvi1.7-10
So far I've done a fair amount of my job. There are eight source package left to get tweaked. Those might be a bit more difficult and require more attention though. What I also did during those efforts: Convert all packages to source format 3.0 (quilt), and use a dh style debian/rules file. The latter enabled the packages to build reproducible too, which is also an added benefit. So this is a win situation on many levels. One of the most prominent reasons why I didn't convert to a dh style debian/rules file yet was that I considered it making easy things easy and difficult things difficult. Finding out what to override and how to do that was something I was unable to figure out, and speaking with people didn't help me there neither. Only recently someone told me that there is dh binary --no-act to figure out what would be called, and then you just prefix it with override_ in debian/rules to get to where you want to go. This worked extremely well for me.
I'm personally still not a big fan of source format 3.0 (quilt) with respect to that it insists on patches to be applied and leaves them that way after building the source package, which makes it difficult to deal with when having upstream source in the VCS too, but I managed to find my way around so many things in the past that I can live with that. The benefit of not having to repack upstream source if it isn't in .gz form is a far bigger benefit. So, I hope to stay productive and be able to get the remaining package also adjusted and fixed. Guess that's doable until the end of the month, and getting rid of all reproducible build bugreports against my packages along that lines. I will check those packages that carry my name already too after my old name is gone from the overview page.

/debian permanent link Comments: 5 Flattr this

12 January 2016

Antoine Beaupr : The Downloadable Internet

How corporations killed the web I have read with fascination what we would have called before a blog post, except it was featured on The Guardian: Iran's blogfather: Facebook, Instagram and Twitter are killing the web The "blogfather" is Hossein Derakshan or h0d3r, an author from Teheran that was jailed for almost a decade for his blogging. The article is very interesting both because it shows how fast things changed in the last few years, technology-wise, but more importantly, how content-free the web have become, where Facebook's last acquisition, Instagram, is not even censored by Iran. Those platforms have stopped being censored, not because of democratic progress but because they have become totally inoffensive (in the case of Iran) or become a tool of surveillance for the government and targeted advertisement for companies (in the case of, well, most of the world). This struck a chord, personally, at the political level: we are losing control of the internet (if we ever had it). The defeat isn't directly political: we have some institutions like ICANN and the IETF that we can still have an effect on, even if only at the technological level. The defeat is economic, and, of course, through economy comes enormous power. That defeat meant that we have first lost free and open access to the internet (yes, dialup used to be free) and then free hosting of our content (no, Google and Facebook are not free, you are the product). This marked a major change in the way content is treated online. H0d3r explains this as the shift from a link-based internet to a stream-based internet, a "deparure from a books-internet towards a television-internet". I have been warning about this "television-internet" in my talks and conversation for a while and with Netflix taking the crown off Youtube (and making you pay for it, of course), we can assuredly say that H0d3r is right and the television, far from disappearing, is finally being resurrected and taking over the internet.

The Downloadable internet and open standards But I would like to add to that: it is not merely that we had "links" before. We had, and still have, open standards. This made the internet "downloadable" (and by extension, uploadable) and decentralized. (In fact, I still remember my earlier days on the web when I would actually download images (as in "right-click" and "Save as..." images, not just have the browser download and display it on the fly). I would download images because they were big! It could take a minute or sometimes more to download images on older modems. Later, I would do the same with music: I would download WAV files before the rise of the MP3 format, of which I ended up building a significant collection (just fair use copies from friends and owned CDs, of course) and eventually video files.) The downloadable internet is what still allows me to type this article in a text editor, without internet access, while reading H0d3r's blog post on my e-reader, because I downloaded his article off an RSS feed. It is what makes it possible for anyone to download a full copy of this blog post and connected web pages as a git repository and this way get the full history of modifications on all the pages, but also be able to edit it offline and push modifications back in. Wikipedia is downloadable (there are even offline apps for your phone). Open standards like RSS feeds and HTML are downloadable. Heck, even the Internet Archive is downloadable (and I mean, all of it, not just the parts you want), surprisingly enough.

The app-based internet and proprietary software App-based websites like Google Plus and Facebook are not really downloadable. They are meant to be browsed through an app, so what you actually see through your web browser is really more an application, downloaded software than a downloaded piece of content. If you turn off Javascript, you will see that visiting Facebook actually shows no content: everything is downloaded on the fly by an application itself downloaded, on the fly, by your browser. In a way, your browser has become an operating system that runs proprietary, untrusted and unverified applications from the web. (The software is generally completely proprietary, except some frameworks that are published as free software in what looks like the lenient act of a godly king, but is actually more an economic decision of a clever corporation which outsources, for free, R&D and testing to the larger free software community. The real "secret sauce" is basically always proprietary, if only so that we don't freak out on stuff like PRISM that reports everything we do to the government.) Technology is political. This new "app design" is not a simple optimization or an cosmetic accident of a fancy engineer: by moving content through an application, Facebook, Twitter and the like can see exactly what you do on a web page, what you actually read (as opposed to what you click on) and how long. By adding a proprietary interface between you and the content online, the advertisement-surveillance complex can track every move you make online. This is a very fine-tuned surveillance system, and because of the App, you cannot escape it. You cannot share the content outside of Facebook, as you can't download it. Or at least, it's not obvious how you can. Projects like youtube-dl are doing an amazing job reverse-engineering what is becoming the proprietary Youtube streaming protocol, which is constantly changing and is not really documented. But it's a hack: it's a Sisyphus struggle which is bound to fail, and it does, all the time, until we figure out how to either turn those corporations into good netizens respecting and contributing to open standards (unlikely) or destroy those corporations (most likely). You are trapped in their walled garden. No wonder internet.org is Facebook only: for most people nowadays, the internet is the web, and the web is Facebook, Twitter and Google, or an iPad with a bunch of apps, each their own cute little walled garden, crafted just for you. If you think you like the Internet, you should really reconsider what you are watching, what you are consuming, or rather, how it is consuming you. There are alternatives. Facebook is a though nut to crack for free software activists because we lack the critical mass. But Facebook it is also an addiction for a lot of people, and spending less time on that spying machine could be a great improvement for you I am sure. For everything else, we have good free software alternatives and open standards, use them. "Big brother ain't watching you, you're watching him." - CRASS, Nineteen Eighty Bore (audio)

Gunnar Wolf: Readying up for the second "Embedded Linux" diploma course

I am happy to share here a project I was a part of during last year, that ended up being a complete success and now stands to be repeated: The diploma course on embedded Linux, taught at Facultad de Ingenier a, UNAM, where I'm teaching my regular classes as well. Back in November, we held the graduation for our first 10 students. This photo shows only seven, as the remaining three have already relocated to Guadalajara, where they were hired by Continental, a company that promoted the creation of this specialization program. After this first excercise, we went over the program and made some adequations; future generations will have a shorter and more focused program (240 instead of 288 hours, leaving out several topics that were not deemed related to the topic or were thoroughly understood by students to begin with); we intend to start the semester-long course in early February. I will soon update here with the full program and promotional material, as soon as I receive it. update (01-19): You can download the promotional information, or go to an (unofficial) URL with the full information. We are close to starting the program, so hurry! I am specially glad that this course is taught by people I admire and recognize, and a very interesting mix between long-time academic and stemming from my free-software-related friends: From the academic side, Facultad de Ingenier a's professors Laura Sandoval, Karen S enz and Oscar Valdez, and from the free-software side, Sandino Araico, Iv n Chavero, C sar Y ez and Gabriel Salda a (and myself on both camps, of course )
AttachmentSize
OT401_nota.jpg75.82 KB
Linux Embebido - Folleto.pdf455.32 KB

11 January 2016

Norbert Preining: 10 years TeX Live in Debian

I recently dug through my history of involvement with TeX (Live), and found out that in January there are a lot of anniversaries I should celebrate: 14 years ago I started building binaries for TeX Live, 11 years ago I proposed the packaging TeX Live for Debian, 10 years ago the TeX Live packages entered Debian. There are other things to celebrate next year (2017), namely the 10 year anniversary of the (not so new anymore) infrastructure in short tlmgr of TeX Live packaging, but this will come later. In this blog post I want to concentrate on my involvement in TeX Live and Debian. TeX Live/Debian Those of you not interested in boring and melancholic look-back onto history can safely skip reading this one. For those a bit interested in the history of TeX in Debian, please read on. Debian releases and TeX systems The TeX system of choice has been for long years teTeX, curated by Thomas Esser. Digging through the Debian Archive and combining it with changelog entries as well as personal experiences since I joined Debian, here is a time line of TeX in Debian, all to my best knowledge.
Date Version Name teTeX/TeX Live Maintainers
1993-96 <1 ? ? Christoph Martin
6/1996 1.1 Buzz ?
12/1996 1.2 Rec ?
6/1997 1.3 Bo teTeX 0.4
7/1998 2.0 Ham teTeX 0.9
3/1999 2.1 Slink teTeX 0.9.9N
8/2000 2.2 Potato teTeX 1.0
7/2002 3.0 Woody teTeX 1.0
6/2005 3.1 Sarge teTeX 2.0 Atsuhito Kohda
4/2007 4.0 Etch teTeX 3.0, TeX Live 2005 Frank K ster, NP
2/2009 5.0 Lenny TeX Live 2007 NP
2/2011 6.0 Squeeze TeX Live 2009
5/2013 7.0 Whezzy TeX Live 2012
4/2015 8.0 Jessie TeX Live 2014
??? ??? Stretch TeX Live 2015
The history of TeX in Debian is thus split more or less in 10 years teTeX, and 10 years TeX Live. While I cannot check back to the origins, my guesses are that already in the very first releases (te)TeX was included. The first release I can confirm (via the Debian archive) shipping teTeX is the release Bo (June 1997). Maintainership during the first 10 years showed some fluctuation: The first years/releases (till about 2002) were dominated by Christoph Martin with Adrian Bunk and few others, who did most packaging work on teTeX version 1. After this Atsuhito Kohda with help from Hilmar Preusse and some people brought teTeX up to version 2, and from 2004 to 2007 Frank K ster, again with help of Hilmar Preusse and some other, took over most of the work on teTeX. Other names appearing throughout the changelog are (incomplete list) Julian Gilbey, Ralf Stubner, LaMont Jones, and C.M Connelly (and many more bug-reporters and fixers). Looking at the above table I have to mention the incredible amount of work that both Atsuhito Kohda and Frank K ster have put into the teTeX packages, and many of their contributions have been carried over into the TeX Live packages. While there haven t been many releases during their maintainership, their work has inspired and supported the packaging of TeX Live to a huge extend. Start of TeX Live I got involved in TeX Live back in 2002 when I started building binaries for the alpha-linux architecture. I can t remember when I first had the idea to package TeX Live for Debian, but here is a time line from my first email to the Debian Developers mailing list concerning TeX Live, to the first accepted upload:
Date Subject/Link Comment
2005-01-11 binaries for different architectures in debian packages The first question concerning packaging TeX Live, about including pre-built binaries
2005-01-25 Debian-TeXlive Proposal II A better proposal, but still including pre-built binaries
2005-05-17 Proposal for a tex-base package Proposal for tex-base, later tex-common, as basis for both teTeX and TeX live packages
2015-06-10 Bug#312897: ITP: texlive ITP bug for TeX Live
2005-09-17 Re: Take over of texinfo/info packages Taking over texinfo which was somehow orphaned started here
2005-11-28 Re: texlive-basic_2005-1_i386.changes REJECTED My answer to the rejection by ftp-master of the first upload. This email sparked a long discussion about packaging and helped improve the naming of packages (but not really the packaging itself).
2006-01-12 Upload of TeX Live 2005-1 to Debian The first successful upload
2006-01-22 Accepted texlive-base 2005-1 (source all) TeX Live packages accepted into Debian/experimental
One can see from the first emails that at that time I didn t have any idea about Debian packaging and proposed to ship the binaries built within the TeX Live system on Debian. What followed was first a long discussion about whether there is any need for just another TeX system. The then maintainer Frank K ster took a clear stance in favor of including TeX Live, and after several rounds of proposals, tests, rejections and improvements, the first successful upload of TeX Live packages to Debian/experimental happened on 12 January 2006, so exactly 10 years ago. Packaging Right from the beginning I used a meta-packaging approach. That is, instead of working directly with the source packages, I wrote (Perl) scripts that generated the source packages from a set of directives. There were several reasons why I choose to introduce this extra layer: Till now I am not 100% sure whether it was the best idea, but the scripts remain in place till now, only adapted to the new packaging paradigm in TeX Live (without xml) and adding new functionality. This allows me to just kick off one script that does do all the work, including building .orig.tar.gz, source packages, binary packages. For those interested to follow the frantic activity during the first years, there is a file CHANGES.packaging which for the years from 2005 to 2011 documents very extensively the changes I made in these years. I don t want to count the hours the went into all this  Development over the years TeX Live 2005 was just another TeX system but not the preferred one in Debian Etch and beyond. But then in May 2006, Thomas Esser announced the end of development of teTeX, which cleared the path for TeX Live as main TeX system in Debian (and the world!). The next release of Debian, Lenny (1/2009), already carried only TeX Live. Unfortunately it was only TeX Live 2007 and not 2008, mostly due to me having been involved in rewriting the upstream infrastructure based on Debian package files instead of the notorious xml files. This took quite a lot of attention and time from Debian away to upstream development, but this will be discussed in a different post. Similarly, the release of TeX Live included in Debian Squeeze (released 2/2011) was only TeX Live 2009 (instead of 2010), but since then (Wheezy and Jessie) the releases of TeX Live in Debian were always the latest released ones. Current status Since about 2013 I am trying to keep a regular schedule of new TeX Live packages every month. These helps me to keep up with the changes in upstream packaging and reduces the load of packaging a new release of TeX Live. It also bring to users of unstable and testing a very up-to-date TeX system, where packages at most lack 1 month of behind the TeX Live net updates. Future As most of the readers here know, besides caring for TeX (Live) and related packages in Debian, I am also responsible for the TeX Live Manager (tlmgr) and most of upstream s infrastructure including network distribution. Thus, my (spare, outside work) time needs to be distributed between all these projects (and some others) which leaves less and less time for Debian packaging. Fortunately the packaging is in a state that making regular updates once a month is less of a burden, since most steps are automatized. What is still a bit of a struggle is adapting the binary package (src:texlive-bin) to new releases. But also this has become simpler due to less invasive changes over the years. All in all, I don t have many plans for TeX Live in Debian besides keeping the current system running as it is. Search for and advise to future maintainers and collaborators I would be more than happy if new collaborators appear, with fresh ideas and some spare time. Unfortunately, my experience over these 10 years with people showing up and proposing changes (anyone remembers the guy proposing a complete rewrite in ML or so?) is that nobody really wants to invest time and energy, but search for quick solutions. This is not something that will work with a package like TeX Live, sized of several gigabyte (the biggest in the Debian archive), and complicated inner workings. I advise everyone being interested in helping to package TeX Live for Debian, to first install normal TeX Live from TUG, get used to what actions happen during updates (format rebuilds, hyphenation patters, map file updates). One does not need to have a perfect understanding of what exactly happens down there in the guts (I didn t have in the beginning, either), but if you want to help packaging and never heard about what format dumps or map files are, then this might be a slight obstacle. Conclusion TeX Live is the only TeX system in wide use across lots of architectures and operating systems, and the only comparable system, MikTeX, is Windows specific (also there are traces of ports to Unix). Backed by all the big user groups of TeX, TeX Live will remain the prime choice for the foreseeable future, and thus also TeX Live in Debian.

Thadeu Lima de Souza Cascardo: GNU on Smartphones (part II)

Some time ago, I wrote how to get GNU on a smartphone. This is going to be a long discussion on why and how we should work on more operating systems for more devices. On Android So, why should we bother if we already have Android, some might ask? If it's just because of some binary blobs, one could just use Replicant, right? Well, one of the problems is that Android development is done in hiding, and pushed downstream when a new version is launched. There is no community behind that anyone can join. Replicant ends up either following it or staying behind. It could do a fork and have its own innovations. And I am all for it. But the lack of manpower for supporting devices and keeping up with the latest versions and security fixes already takes most of the time for the one or two developers involved. Also, Android has a huge modularity problem, that I will discuss further below. But it's hard to replace many components in the system, unless you replace them all. And that also causes the problem that applications can hardly share extra components. I would rather see Debian running on my devices and supporting good environments and frameworks for all kinds of devices, like phones, tablets, TVs, cars, etc. It's developed by a community I can join, it allows a diverse set of frameworks and environments, and it's much easier to replace single components on such a system. On Copyleft Hardware I get it. Hardware is not free or libre. Its design is. I love the concept of copyleft hardware, where one applies the copyleft principles to a hardware design. Also, there is the concept of Hardware that Respects Your Freedom, that is, one that allows you to use it with only free software. Note that RYF hardware is not necessarily copyleft hardware. In fact, most of the time, the original vendor has not helped at all, and it required reverse engineering efforts by other people to be able to run free software on those systems. My point here is that we should not prevent people from running free software on hardware that does not RYF or that is not copyleft. We should continue the efforts of reverse engineering and of pushing hardware vendors to not only ship free software for their hardware, but also release their design under free licenses. But in the meantime, we need to make free software flourish in the plethora of devices on the hands of so many people around the world. On Diversity I won't go into details on this post about two things. One topic I love is retrocomputing and how Linux supported so many devices, and how many free or quasi-free operating systems ran on so many of them in the past. I could mention ucLinux, Familiar, PDAs, Palmtops, Motorolas, etc. But I will leave it to another time and go from Openmoko and Maemo forward. The other topic is application scalability. Even Debian with so many software available does not ship all free software there is available. And it doesn't support all third-party services out there. How can we solve that? It has to do with platforms, protocols, open protocols, etc. I will not go into that today. Because I believe that either way, it's healthy for our society to have diversity. I believe we should have other operating systems available for our devices. Even if application developers will not develop for all of them. That is already the case today. There are other ways to fix that, when that needs fixing. Sometimes, it's sufficient that you can have your own operating system on your device, that you can customize it, enhance it and share it with friends. And also, that would allow for innovation. It would make possible that some other operating system would have enough market share on some other niche. Other than Android and iOS, for example. But that requires that we can support that operating system on different devices. And that is the scalability that I want to talk about. How to support more devices with less effort. Options But before I go into that, let me write more about the alternatives we have out there. And some of the history around it. Openmoko Openmoko developed a set of devices that had some free design. And it has some free operating systems running on top. The community developed a lot of their own as well. Debian could (still can) run on it. There is SHR, which uses FSO, a mobile middleware based on D-Bus. It even spawned other devices to be developed, like GTA04, a successor board, that can be shipped inside a Neo Freerunner case. Maemo, Moblin, Meego and Tizen I remember the announcement of Nokia N770. During FISL in 2005, I even criticized a lot, because it shipped with proprietary drivers. And applications. But it was the first time we heard of Maemo. It was based on Debian and GNOME. The GNOME Mobile initiative was born from that, I believe, but died later on. But with the launch of N900, and later events, like the merge of Moblin with Maemo, to create Meego, we all had an operating system that was based on community developed components, that had some community participation, and was much more like the systems we were used to. You could run gcc on N900. You could install Das U-Boot and have other operating systems running there. But Nokia has gone a different path. Intel has started Tizen with Samsung. There is so much legacy there, that could still be developed upon. I am just sad that Jolla decided to put proprietary layers on top of that, to create SailfishOS. But we still have Mer, Nemo. But it looks like Cordia is gone. At least, (http://cordiahd.org) seems to have been registered by a domain seller. Not to forget, Neo900, a project to upgrade the board, in the same vein as GTA04. FirefoxOS and Ubuntu What can I say about FirefoxOS and Ubuntu Phone? In summary, I think we need more than Web, and Canonical has a history of not being too community oriented as we'd like. I won't go too much here in what I think about the Web as a platform. But I think we need a system that has more options for platforms. I haven't participated in projects directed by Mozilla either, so I can't say much about that. Ubuntu Phone should be a system more like what we are used to. But Canonical is going to set the directions to the project, it's not a community project. Nonetheless, I think they add to the diversity, and users should be able to try them, or their free parts or versions. But there are challenges there, that I think need to be discussed. So, both of these systems are based on Android. They don't use Dalvik or the application framework. But they use what is below that, which means device drivers in userspaces, like RIL, for the modem drivers, and EGL, for graphics drivers. The reason they do that is to build on top of all the existing vendor support for Android. If a SOC and phone vendor already supports Android, there should be not much needed to do to support FirefoxOS or Ubuntu Phone. In practice, this has not benefited the users or the projects. Porting should be as simple as getting a generic FirefoxOS or Ubuntu Phone image and run it on top of your Android or CyanogenMod port. Porting usually requires doing all the same work as porting Android. Even though one should be able to take advantage of existing ports, it still requires doing a lot of integration work and building an entire image, most of the time, including the upper layers, that should be equal in all devices. This process should require at most creating a new image based on other images and loading it on the device. I will discuss more about this below. I can't forget to mention the Matchstick TV project. It is based on FirefoxOS. I think it would have much more changes to succeed if it was easier for testers to have images available for all of their gadgets capable of running some form of Android. Replicant And then we have Replicant. It has a lot of the problems Android has. Even so, it's a tremendously important project. First, it offers a free version of Android, removing all proprietary pieces. This is very good for the community. But more than that, it tries to free the devices beyond that. What the project has been done is reverse engineer some of the pieces, mostly the modem drivers. That allows other projects to build upon that work, and support those modems. Without that, there is no telephony or celullar data available on any of these devices. Not without proprietary software, that is. They also have been working on free bootloaders, which is another important step for a completely free system. Next steps There are many challenges here, but I believe we should work on a set of steps that make it more palatable and produce intermediary results that can be used by users and other projects. One of the goals should be good modularity. The ability to replace some pieces with the least trouble necessary. The update of a single framework library should be just a package install, instead of building the entire framework again. If there is ABI breakage, users should still be able to have access to binaries (and accompanying source code) and only need to update the library and its reverse dependencies. If one layer does not have these properties, at least it should be possible to update this big chunk without interfering with the other layers. One example is FirefoxOS and Ubuntu Phone. Even if there is a big image with all system userspace and middleware, the porting, install and upgrade process should allow the user to keep the applications and to leverage porting already done by similar projects. Heck, these two projects should be able to share porting efforts without any trouble. So what follows is a quick discussion on Android builds, then suggestions on how to approach some of the challenges. On Android build The big problem with Android build is its non-modularity. You need to build the world and more to get a single image in the end. Instead of packages that could be built independently, that generated modular package images, that could be bundled in a single system image. Better yet would be the ability to pick only those components that matter. Certainly, portability would be just as interesting, being able to build certain components on top a GNU system with GNU libc. At times, it seems like this is done by design, to make it difficult for "fragmentation" and competition. Basically, making it difficult to exercise the freedom to modify the software and share your modifications with others. Imagine the possibilities of being able to: There are some dangers in promoting the use of Android like this. Since it promotes proprietary drivers, relying on such layers instead of better community oriented layers means giving an advantage to the first. So, the best plan would be to replace those layers, when they are used, for things like ofono and Wayland, for example. But, in the meantime, making it easier for users to experiment with FirefoxOS, Ubuntu Phone, Matchstick TV, Replicant on other devices, without resorting to building the entire Android system, would be a very nice thing. It is possible that there are some challenges with respect to API and ABI. That these layers are too intermingled that some changes present in one port would prevent FirefoxOS to run on top of it without changes in either of the layers. I can't confirm that is the case, but can't deny the possibility. Rooting One of the challenges we have that may have trouble in scaling is rooting devices. Unfortunately, most of the devices are owned by the vendor or carrier, not the user. The user is prevented from replacing software components in the system, by removing its permissions from replacing most of the files in the filesystem, or writing to block devices, or change boot parameters or write to the storage devices. Fortunately, there are many documents, guides, howtos, and whatnots out there instructing on how to root a lot of devices. In some cases, it depends on software bugs that can be patched by the users as instructed by vendors. Certainly, favoring devices that are built to allow rooting, hacking, etc, is a good thing. But we should still help those users out there who do not have such devices, and allow them to run free software on them. Booting Then, comes booting, which is a large discussion on its own, and also related to rooting, or how to allow users to replace their software. First, we have the topic of bootloaders, which are usually not free, and embedded in the chip, not on external storage. So, there are those pieces of the bootloader which we can replace more easily and those that we are not, because they would require changing a piece of ROM, for example. Das U-Boot would be one of the preferred options for a system. It supports a lot of hardware platforms out there, a lot of storage devices and filesystems, network booting, USB and serial booting, and a lot of other things. Porting is not an easy task, but it is possible. Meanwhile, using the bootloaders that ship with the system, when possible, would allow a path where other pieces of the system would be free. One of the challenges here is building single images that could boot everywhere. The lack of a single ARM boot environment is a curse and a blessing. It makes it hard to have this single boot image, but, on the other hand, it has allowed so much of the information for such systems to be free, to be embedded in copyleft software, instead of having blobs executed by our free systems, like ACPI encourages. Device trees have pushed this portability issue from the kernel to the bootloaders. Possibily encouraging vendors to now hide and copyright this information in the form of proprietary device trees. But it has made it easier for single images. In practice, we still need to see a lot of devices out there supporting this single kernel scenario. And this mechanism only brings us the possibility of a single kernel. We still need to ship bootable images that have pieces of bootloaders that are not as portable. This has caused lots of operating systems out there to be built for a single board. Or to support just a small set of boards. I struggle with the concept. Why are we not able to mix device-specific pieces with generic pieces and get a resulting image that will boot on our choice of board? Why does every project need to do all the porting work again, repeating the efforts? Or have one entier ISO image for every supported board? Check how Geexbox does it, for an example. Fortunately, I see Debian going in a good direction. Here, one can see how it instructs the user to combine a system-dependent part with a system-independent part. Installation Which brings us to the installation challenge. We should make this easy and also customizable by the user. Every project might have its own method, and that is part of the diversity that we should allow. The great challenge here is handling rooting and booting for the user. But it's also possible to leave that as separate efforts, as it would be nice to have good installers as we have in the desktop and server environments. Linux Linux is one of the most ported kernels out there. Of course, it would be healthy to the diversity I propose that other kernels and systems out there should work on those mobile devices. NetBSD, for example, has a reputation of being very portable and ported to many platforms. And GNU runs on many of them, either together with the original systems, or as the main system on top of other kernels, as proven by the Debian GNU/kFreeBSD port, which uses GNU libc. But, though I would love to see HURD running directly on those devices, Linux and its derivatives are already running on them. Easier to tackle is to get Linux-libre running on those systems. But though this looks like one of the easier tasks at hand, it is not. If you run a derivative version from Linux, provided by the vendor, things should go smooth. But most of the time, the vendor does not provide patches upstream, and leave their fork to rot, if you are lucky, in the earlier 3.x series. And some times, there is not even source code available. There are vendors out there who do not respect the GPL and that is one of the reasons why GPL enforcement is important. I take this opportunity to call attention to the work of Software Freedom Conservancy and its current efforts to raise funds to continue their work. Running an existing binary version of Linux on your device with a free userspace is part of the strategy of replacing pieces one at a time while allowing for good diversity and reaching more users. Drivers Then we have the matter of drivers. And in this case, it's not only Linux drivers, but drivers running in userspace. Though others exist, there are two important and common cases, which are modem and graphics drivers, both of which are usually provided by Android frameworks, and that other systems try to leverage, instead of using alternatives that are more community friendly. In the case of modem drivers, Android uses its RIL. Besides not using a Radio or Modem at all, there are two strategies for free versions. One is using free versions of the drivers for RIL. That's what Replicant does, because, well, it uses the other Android subsystems after all. The other one is using a system that is developed by the community. ofono is one that, even though it's an Intel iniative, looks more community governed or open to community participation than any Android subsystem ever was. As for Graphics, Canonical even built its Mir project with the goal of being able to use Android GPU drivers. Luckily, there has been a lot of reverse engineering efforts out there for a lot of those GPUs shipped with ARM SoCs. Then, we can use Mesa, Wayland and X.org. Other option, until then, is just using a framebuffer driver, possibly one that does not need any initialization, depending on one done by the bootloader, for example. Middleware and Shells Plasma Active, that I just found out now is Plasma Mobile, looks like a great system. Built by folks behind KDE, we can expect great freedom to come from it. Unfortunately, it suffers from what we have been discussing here, which is lack of good device support, or shipping images that run on top of single devices, without leveraging the porting that has come before of other systems. Fortunately, that is just what I want to accomplish with this effort. FSO, that I mentioned before, is a good middleware, that we should try to run on top of these devices. Running GNOME or Unity shell, and use rpm or deb based systems, that's all part of the plan on diversity. You could use systemd, or System V init systems, whatever gives you the kicks. It's not an easy task, since there are so many new things on these new kinds of devices. Besides touch, there is telephony, which as I mentioned, would be a good task for ofono. As for TV sets or dongles, I would love to see OpenFlint, created by the Matchstick TV guys, flourish out there and allow me to flint stuff from my mobile phone running Debian into my TV dongle running Debian. Project So, is there a project out there you can start contributing to? Well, I pointed out a lot of them. All of them make part of this plan. Contributing to reverse engineering GPUs, or to Plasma Mobile, ofono, GNOME, Linux-libre, bootloaders, Debian, and so many other projects. My plans are to contribute in the scope of Debian and make sure it works well on top of semi-liberated devices, and make sure there is a nice user interface and good applications when using either GNOME or Plasma. Integrating ofono is a good next step, but I am running ahead of myself. Right now, I don't think there is need for an integrated effort. If you think otherwise, please let me know. If you are doing something in this direction, I would love to know. Paul Wise reminded me to point out to the Debian Mobile, where I will probably contribute any documentation and other relevant results. Thanks There are some many people to thank, but I would like to remember Ian Murdock, who created Debian, one of the projects who inspires me the most. I think the best way to handle his passing is to celebrate what he has helped create, and move forward, taking more free software to more people in an easy and modular way. I have been wanting to write something like this for a long time, so I thank Chris Webber for inspiring me on doing it.

Daniel Pocock: FOSDEM RTC Dev-room schedule published

If you want to help make Free Real-time Communication (RTC) with free, open source software surpass proprietary solutions this year, a great place to start is the FOSDEM RTC dev-room. On Friday we published the list of 17 talks accepted in the dev-room (times are still provisional until the FOSDEM schedule is printed). They include a range of topics, including SIP, XMPP, WebRTC and peer-to-peer Real-time communication. RTC will be very prominent at FOSDEM this year with several talks on this topic, including my own, in the main track.

Ben Hutchings: Debian LTS work, December 2015

In December I carried over 15 hours from October/November and was assigned another 15 hours of work by Freexian's Debian LTS initiative. I worked a total of 20 hours despite the holidays. I uploaded a security and bug fix update to linux-2.6 early in December, and sent DLA-360-1. I also backported several more security fixes, released in the new year. I sent several of the fixes to Willy Tarreau for inclusion in Linux 2.6.32-longterm. I prepared an update to sudo to fix CVE-2015-5602. This turned out not to have been properly fixed upstream, so I finished the job and am now in the process of backporting and uploading fixes for all suites. I reviewed the packages affected by CVE-2015-8614 and the upstream fix in claws-mail, and found that that was also incomplete. This resulted in another CVE ID being issued. I had another week in the front desk role, over the new year, and triaged about 20 new issues. About half of them affected packages supported in squeeze-lts. Updated: I also found a bug in the contact-maintainers script used by the LTS front desk. It used apt-cache show to find out the maintainers of a source package, which may result in outdated information particularly if you configure APT to fetch squeeze sources in order to work on LTS! I modified the script to grab maintainer information out of the RDF description provided by packages.qa.debian.org (not yet implemented on tracker.debian.org). I feel there ought to be an easier way to do this, but at least I learned something about RDF.

10 January 2016

Andrew Shadura: Public transport map of Managua

Holger Levsen writes about the public transport map of Managua, Nicaragua, which is, according to him, the first detailed map of Managua s bus network:
If you haven t been to Managua, you might not be able to immediatly appreciate the usefulness of this. Up until now, there has been no map nor timetable for the bus system, which as you can see now easily and from far away, is actually quite big and is used by 80% of the population in a city, where the streets still have no names.
Having had a look at the map they produced, I have to admit I quite liked it: MapaNica.net, Rutas de Managua y Ciudad Sandino MapaNica, the community behind said map, are raising funds to make lives of locals easier by publishing a printed version of the map and distributing it. They have already raised more than $3300 of their $7500 goal. Every further donation will help them print more maps. Please go to support.mapanica.net and support their initiative!

Holger Levsen: 20160110-support-mapanica-net

Please support the public transportation map of Manugua Some of you might remember DebConf12 in Managua, Nicaragua and the very friendly and helpful locals, who recently contacted me to tell me about their new project, so that I share this on planet Debian: A local community of openstreet map enthusiasts, of which some were involved in DebConf12, has collected for the first time detailed information about Managua's bus network! To bring their efforts further, they will now print these maps on paper, so that even more people can use them in their daily lives. If you haven't been to Managua, you might not be able to immediatly appreciate the usefulness of this. Up until now, there has been no map nor timetable for the bus system, which as you can see now easily and from far away, is actually quite big and is used by 80% of the population in a city, where the streets still have no names. If this made you curious (or just brought back happy memories from 2012) please go to http://support.mapanica.net and donate some money - their campaign is running for 3 more weeks and currently they have already raised 3300 USD, enough to print some maps, but 4200 USD short of their goal. Every further donation will help to print some more maps, even something as little as 20 USD or EUR will help people in their real lives to better understand the beast of Managua's bus route network.

Juliana Louback: NLP: Viterbi Named-Entity Tagger

During my MSc program, I was lucky to squeeze into Michael Collin s NLP class. We used his Coursera course as part of the program, which I d highly recommend. Recently I decided to review my NLP studies and I believe the best way to learn or relearn a subject is to teach it. This is one in a series of 4 posts with a walk-through of the algorithms we implemented during the course. I ll provide links to my code hosted on Github. Disclaimer: Before taking this NLP course, the only thing I knew about Python was that it s the one without curly brackets . I learned Python on the go while implementing these algorithms. So if I did anything against Python code conventions or flat-out heinous, I apologize and thank you in advance for your understanding. Feel free to write and let me know.

The Concept To quote Wikipedia, Named-entity recognition (I ve always known it as tagging) is a subtask of information extraction that seeks to locate and classify elements in text into predefined categories such as the names of persons, organizations, locations, expressions of times, quanitites, monetary values, percentages, etc. For example, the algorithm receives as input some text Bill Gates founded Microsoft in 1975. and outputs Bill Gates[person] founded Microsoft[organization] in 1975[date]. Off the top of my head, some useful applications are document matching (ex. a document containing Gates[person] may not be on the same topic as one containing gates[object]) and query searches. I m sure there are lots more, if you check out Collin s Coursera course he may discuss this in greater depth.

The Requirements Development data: The file ner_dev.dat provided by prof. Michael Collins has a series of sentences separated by an empty line, one word per line. Training data: The file ner_train.dat provided by prof. Michael Collins has a series of sentences separated by an empty line, one word and tag per line, speparated by a space. Word-tag count data: The file ner.counts has the format [count] [type of tag] [label] [word]. The tags used are RARE, O, I-MISC, I-PER, I-ORG, I-LOC, B-MISC, B-PER, B-ORG, B-LOC. The tag O means it s not an NE. This file is generated by count_freqs.py, a script provided by prof. Michael Collins. Run count_freqs.py on the training data ner_train.dat

The Algorithm Python code: viterbi.py Usage: python viterbi.py ner.counts ngram.counts [input_file] > [output_file] Summary: The Viterbi algorithm finds the maximum probability path for a series of observations, based on emission and transition probabilities. In a Markov Process, emission is the probability of an output given a state and transition is the probability of transitioning to the state given the previous states. In our case, the emission parameter e(x y) = the probability of the word being x given you attributed tag y. If your training data had 100 counts of person tags, one of which is the word London (I know a guy who named his kid London), e( London person ) = 0.01. Now with 50 counts of location tags, 5 of which are London , e( London location ) = 0.1 which clearly trumps 0.01. The transition parameter q(yi yi-1, yi-2) = the probability of putting tag y in position i given it s two previous tags. This is calculated by Count(trigram)/Count(bigram). For each word in the development data, he Viterbi algorithm will associate a score for a word-tag combo based on the emission and transition parameters it obtained from the training data. It does this for every possible tag and sees which is more likely. Clearly this won t be 100% correct as natural language is unpredictable, but you should get pretty high accuracy. Optional Preprocessing Re-label words in training data with frequency < 5 as RARE - This isn t required, but useful. Re-run count_freqs.py if used. Python code: label_rare.py Usage: python label_rare.py [input_file] Pseudocode:
  1. Uses Python Counter to obtain word counts in [input_file]; removes all word-count pairs with count < 5, store remaining pairs in a dictionary named rare_words.
  2. Iterates through each line in [input file], checks if word is in rare words dictionary, if so, replaces word with RARE.
Step 1. Get Count(y) and Count(x~y) Python code: emission_counts.py Pseudocode:
  1. Iterate through each line in ner.counts file and store each word-label-count combo in a dictionary count_xy and update the dictionary of count_y. For example count_xy[Peter][I-PER] returns the number of times the word Peter was labeled I-PER in the training data and count_y[I-PER] the total number of I-PER tags. The dictionary count_y contains 8 items, one for each label ( RARE , O, I-MISC, I-PER, I-ORG, I-LOC, B-MISC, B-PER, B-ORG, B-LOC);
  2. Return count_xy, count_y
Step 2. Get bigram and trigram counts Python code: transition_counts.py Pseudocode:
  1. Iterate through each line in the n-gram_counts file
  2. If the line contains 2-GRAM add an item to the bigram_counts dictionary using the bigram (two space-separated labels following the tag type 2-gram ) as key, count as value. This dictionary will contain Count(yi-2,yi-1).
  3. If the line contains 3-GRAM , add an item to the trigram_counts dictionary using the trigram as key, count as value. This dictionary will contain Count(yi-2, yi-1, yi).
  4. Return dictionaries of bigram and trigram counts.
Step 3. Viterbi (For each line in the [input_file]):
  1. If the word was seen in training data (present in the count_xy dictionary), for each of the possible labels for the word:
  2. Calculate emission = count_xy[word][label] / float(count_y[label]
  3. Calculate transition = trigram_counts[trigram])/float(bigram_counts[bigram] Note: yi-2 = *, yi-1 = * for the first round
  4. Set probability = emission x transition
  5. Update max(probability) and arg max if needed. 2 If the word was not seen in the training data:
  6. Calculate emission = count xy[RARE][label] / float(count y[label].
  7. Calculate q(yi yi-1, yi-2) = trigram counts[trigram])/float(bigram counts[bigram]. Note: yi-2 = , yi-1 = for the first round
  8. Set probability = emission transition
  9. Update max(probability) if needed, arg max = RARE
  10. Write arg max and log(max(probability)) to output file.
  11. Update yi-2, yi-1.
  12. Update yi-2, yi-1.
Evaluation Prof. Michael Collins provided an evaluation script eval_ne_tagger.py to verify the output of your Viterbi implementation. Usage: python eval_ne_tagger.py ner_dev.key [output_file]

Alessio Treglia: A WordPress Plugin to list posts in complex nested websites

List all posts by Authors, nested Categories and Titles is a WordPress Plugin I wrote to fix a menu issue I had during a complex website development. It has been included in the official WordPress Plugin repository. The Plugin is particularly suitable to all multi-nested categories and multi-authors websites handling a large number of posts and complex nested category layout (i.e.: academic papers, newpapers articles, etc). This plugin allows the user to place a shortcode into any page and get rid of a long and nested menu/submenu to show all site s posts. A selector in the page will allow the reader to select grouping by Category/Author/Title. You can also manage to install a tab plugin (i.e.: Tabby Responsive Tabs) and arrange each group on its specific tab. Output grouped by Category will look like:

CAT1
    post1                       AUTHOR
    SUBCAT1
        post2                   AUTHOR
        post3                   AUTHOR
        SUBCAT2
            post4               AUTHOR
            ...
            ...
while in the Author grouping mode, it is:
AUTHOR1
  post1               [CATEGORY]
  post2               [CATEGORY]
AUTHOR2
  post1               [CATEGORY]
  post2               [CATEGORY]
.....
The plugin installs a new menu ACT List Shortcodes in Admin->Tools. The tool is a helper to automatically generate the required shortcode. It will parse the options and display the string to be copied and pasted into any page. The Plugin is holding a GPL2 license and it can be downloaded from its page on WP Plugins. wordpress-logo

Next.

Previous.