Search Results: "tar"

26 May 2010

Nico Golde: fail2ban + dns = fail

fail2ban is used by many people to prevent certain types of DoS attacks. I use it myself to reduce trackback spam a little bit.

While this tool becomes quite handy in such situations it is also not generally recommend because you can shoot yourself in the foot. If one of the used filters has a bug and results in incorrect parsing your fail2ban installation might end up banning arbitrary IP addresses or even your own IP range (not even mentioning IP spoofing).
There existed at least two bugs of this kind to my knowledge and since regex might not always be easy I'm sure there will be more in the future.

Since I didn't want to look for a specific regex bug in one of the filters I thought about IP spoofing again and looked at fail2bans filters. What I needed was a filter processing log entries of a service listening on a UDP socket as TCP/IP spoofing over the internet doesn't really work well these days. Finding such a filter would mean an instant win situation. To my surprise there is such a filter: config/filter.d/named.conf

This filter is used to parse log entries consisting of denied DNS queries produced by bind. Interestingly there is even an article at debian-administration describing how to setup fail2ban to mitigate a DNS DDoS attack. This is of course a bad idea and I have no idea why this filter is shipped in a default fail2ban installation. DoSing abritary IP addresses with this filter in use becomes as easy as firing up scapy and querying the server with a forged source IP:

>>> send(IP(dst="81.169.172.197",src="xx.46.63.71")/UDP()/DNS(rd=1,qd=DNSQR(qname="foao.modprobe.de")))
.
Sent 1 packets.

This ends up as:
May 26 22:32:22 modprobe named[30245]: client xx.46.63.71#53: query 'foao.modprobe.de/A/IN' denied

in the bind logs which in turn results in:
2010-05-26 22:32:05,551 fail2ban.actions: WARNING [named-refused] Ban xx.46.63.71

In this example the spoofed IP was xx.46.63.71 which is not under my control.

Mission statement: don't use fail2ban unless you really want to shoot yourself in the foot or know pretty well what you're doing :-)

Adrian von Bidder: The Debian Umbrella: Soon.

Update: I'm taking orders now. I was planning to take orders for the Debian Swirl umbrella starting tonight, but a bit of research showed that shipping prices for parcels are insane (CHF 37 for europe), but it seems I can get away with sending this as a Swiss Post Maxi Letter instead, which would result in a shipping price of CHF 12.50 for four and CHF 24 for up to eight umbrellas (1kg or 2kg.) So now I'll go and look into getting some kind of flattish box to stay within the allowed dimensions before I can be sure this is possible. In any case, I expect the delivery in the first week of June. Since I'll not be coming to Debconf, it would be nice if somebody could take some umbrellas to New York. I'm currently not travelling much, so I'd have to rely on a friendly volunteer. (I'm based in Basel, Switzerland; I am known to go to Z rich regularly.)

Matt Zimmerman: Using Mumble with a bluetooth headset

At Canonical, we ve started experimenting with Mumble as an alternative to telephone calls for real-time conversations. The operating model is very much like IRC, based on channels within which everyone can hear everyone else as they speak. Mumble works best with a headset, which offers better audio recording quality due to the proximity of the microphone, and avoids problems with echo and feedback. I like to pace around while I talk, and so I ve already invested in a Plantronics Calisto Pro, which includes a DECT handset, a Bluetooth headset and a nice charging base. My laptop has bluetooth onboard, so I set about trying to get Mumble set up to use the headset via bluetooth. The first thing I tried was to click on the bluetooth icon on the panel, and select Set up new device.... After setting the headset to pairing mode, I waited quite some time for it to show up in the list, but it never did. After opening the preferences dialog, I discovered that this was (presumably) because I had already paired it, ages ago. So, I went about trying to get PulseAudio to talk to it. After some hunting, I tried:
pactl load-module module-bluetooth-device address=00:23:xx:xx:xx:xx
This created a new Card in PulseAudio, which I could see in the Hardware tab of the Sound Preferences dialog and in pactl list, but it was inactive:
Card #1
Name: bluez_card.00_23_XX_XX_XX_XX
Driver: module-bluetooth-device.c
Owner Module: 17
Properties:
device.description = "Calisto PLT"
device.string = "00:23:XX:XX:XX:XX"
device.api = "bluez"
device.class = "sound"
device.bus = "bluetooth"
device.form_factor = "headset"
bluez.path = "/org/bluez/13899/hci0/dev_00_23_XX_XX_XX_XX"
bluez.class = "0x200404"
bluez.name = "Calisto PLT"
device.icon_name = "audio-headset-bluetooth"
Profiles:
hsp: Telephony Duplex (HSP/HFP) (sinks: 1, sources: 1, priority. 20)
off: Off (sinks: 0, sources: 0, priority. 0)
Active Profile: off
The Hardware tab confirmed that the device was Disabled, and using the Off profile. I could manually select the Telephony Duplex (HSP/HFP) profile, but this had no apparent effect. There were no Sources or Sinks to send or receive audio data to or from the headset (and thus nothing new in the Input or Output tabs of the preferences dialog). syslog hinted:
pulseaudio[15239]: module-bluetooth-device.c: Default profile not connected, selecting off profile
At this point, I recalled that when I suspend this laptop, the bluetooth driver gets unhappy. I don t commonly use bluetooth on this laptop, so I hypothesized that the driver was in a weird state, and I decided to try unloading and reloading the btusb module. Once I did so, the device showed up in the panel menu, with a Connect menu item. Aha! The manual module loading above may turn out to be unnecessary if the device shows up in the menu initially. I selected the Connect menu item, and a bunch of magic happened, with the result that I heard the headset s tone to indicate it was activated. Sound Preferences now showed it under Hardware as active using the Telephony Duplex profile, and it appeared under Input and Output as well. pactl list showed its sources and sinks. Mumble offered it as a choice for the input and output device. Progress! Some experimentation (thanks, Colin) revealed that other people could hear me through the headset just fine. However (Problem #1), I couldn t hear them clearly through the headset. If I switch Mumble s output to my speakers, they sound fine, so it s not Mumble. So I tried:
paplay -d bluez_sink.00_23_XX_XX_XX_XX /usr/share/sounds/alsa/Front_Center.wav
which also sounds awful. There is a whining noise, which gets louder when the audio signal is louder, which makes it very difficult to hear. I don t know if the problem is with PulseAudio, bluez, the kernel, or the device, but using speakers for output is a workaround. This does seem to cause some echo, though, so I ll need to track this down eventually. Problem #2 is that Mumble seems to prevent PulseAudio from suspending the headset s source. Even if I set it to Push to Talk mode, the headset stays active all the time, which will drain the battery. PulseAudio seems to do the right thing, and kill the radio link, if the source is left idle, and brings it up when there is activity, so this looks to be Mumble s fault. I ll need to fix this as well.

Russell Coker: UBAC and SE Linux in Debian

A recent development in SE Linux policy is the concept of UBAC (User Based Access Control) which prevents SE Linux users (identitied) from accessing each other s files. SE Linux user identities may map 1:1 to Unix users (as was required in the early versions of SE Linux), you might have unique identities for special users and a default identity for all the other users, or you might have an identity per group or use some other method of assigning identities to groups. The UBAC constraints in the upstream reference policy prevent a process with a SE Linux identity other than system_u from accessing any files with an identity other than system_u. So basically any regular user can access files from the system but not other users and system processes (daemons) can access files from all users. Of course this is just one layer of protection, so while the UBAC constraint doesn t prevent a user from accessing any system files the domain-type access controls may do so. If you used a unique SE Linux identity for each Unix account then UBAC would prevent any user from accessing a file created by another user. For my current policy that I am considering uploading to Debian/Unstable I have allowed the identity unconfined_u to access files owned by all identities. This means that unconfined_u is an identity for administrators, if I proceed on this path then I will grant the same rights to sysadm_u. UBAC was not enabled in Fedora last time I checked, so I m wondering whether there is any point in including it I don t feel obliged to copy everything that Fedora does, but there is some benefit in maintaining compatibility across distributions. For protecting users from each other it seems that MCS (which is Mandatory in the Debian policy) is adequate. MCS allows a much better level of access control. For example I could assign categories c0 to c10 to a set of different projects and allow the person who manages all the projects to be assigned all those categories when they login. That user could then use the command runcon -l s0:c1 bash to start a shell for the purpose of managing files from project 1 and any file or process created by that command would have the category c1 and be prevented from writing to a file with a different category. Of course the down-side to removing UBAC is that since RBAC was removed there is no other way of separating SE Linux users, while MCS is good for what it does it wasn t designed for the purpose of isolating different types of user. So I ll really want to get RBAC reinstalled before Squeeze is released if I remove UBAC. Regardless of this I will need to get RBAC working on Squeeze eventually anyway. I ve had a SE Linux Play Machine running with every release of SE Linux for the last 8 years and I don t plan to stop now.

Jose Luis Rivas Contreras: Chrome vs Firefox, again

Long time ago I stopped using Chrome for daily use and leave it only for my Google services like Gmail and Wave. Now, is my main browser even when it just stop working while I m offline (weird thing, btw). Is fast, starts fast, loads fast, runs javascript very fast. I can use websites as apps without installing another app to complement my browser like Prism and renders fonts wonderfully with @font-face. Now, Firefox, well First of all, I m using nightly builds of alpha version, so it s supposed to fail sometimes and I understand that but, it s not supposed to be faster as well? Loads after 1 min I launch it(UPDATE 2010-05-26: There was a loop on an addon making it start slow, now starts on 10 seconds which is pretty nice) and loads not as smooth as Chrome the websites, even using NoFlash, NoAds and NoJavaScript! In fact, if wasn t because Chrome stops working while I m offline I would not have any reason to use Firefox even while designing websites (just for checking). It s a shame, I really liked a lot Firefox and I hope Firefox4 get a really good speed bump.

25 May 2010

Erich Schubert: Facebook and privacy

Mark Zuckerberg apparently recently promised
we will add privacy controls that are much simpler to use.
Dear Facebook. I believe your users want more privacy, not just prettier controls. They don't want you to give away their data by default, track them across web sites (that embed the "Like" buttons) and auto-connect them to various things.In particular, they want opt-in, not opt-out. And they also don't want to be auto-opted in by you either. Make the defaults all opt-out!They don't want you to index and publish their data by default like if you were a paparazzi and they were a pop star. Sure, they'd like to be a pop star, but without the paparazzi stuff, you know.There is an essential difference between actually caring about privacy, and just making it look nice (which unfortunately is what Facebook currently does, pretend it's all fine, because you could turn it off, if you go to all the sites you don't want and turn it off on each single one ...).

Aurelien Jarno: EGLIBC and PowerPCSPE port

This has been roughly one year since Debian switched from GLIBC to EGLIBC, so it s probably the time to make a small report about this change. First of all, on the GLIBC upstream side, things has improved a bit since we now have regular stable release, thanks to Petr Baudis aka Pasky. The good point is that the stable releases are imported into the EGLIBC stable repositories. On the EGLIBC side the switch has helped to reduce the number of patches in the Debian package (for example, resolv.conf is automatically reloaded if needed), and has brought some bug fixes and improvements, especially for the arm, mips and powerpc targets. It should be noted that the newly created PowerPCSPE port for PowerPC e500 series CPU also benefits from EGLIBC, as it is not natively supported by GLIBC.

Matt Zimmerman: The behavioral economics of free software

People who use and promote free software cite various reasons for their choice, but do those reasons tell the whole story? If, as a community, we want free software to continue to grow in popularity, especially in the mainstream, we should understand better the true reasons for choosing it especially our own. Some believe that it offers higher quality, that the availability of source code results in a better product with higher reliability. Although it s difficult to do an apples-to-apples comparison of software, there are certainly instances where free software components have been judged superior to their proprietary counterparts. I m not aware of any comprehensive analysis of the general case, though, and there is plenty of anecdotal evidence on both sides of the debate. Others prefer it for humanitarian reasons, because it s better for society or brings us closer to the world we want to live in. These are more difficult to analyze objectively, as they are closely linked to the individual, their circumstances and their belief system. For developers, a popular reason is the possibility of modifying the software to suit their needs, as enshrined in the Free Software Foundation s freedom 1. This is reasonable enough, though the practical value of this opportunity will vary greatly depending on the software and circumstances. The list goes on: cost savings, educational benefits, universal availability, social rewards, etc. The wealth of evidence of cognitive bias indicates that we should not take these preferences at face value. Not only are human choices seldom rational, they are rarely well understood even by the human themselves. When asked to explain our preferences, we often have a ready answer indeed, we may never run out of reasons but they may not withstand analysis. We have many different ways of fooling ourselves with regard to our own past decisions and held beliefs, as well as those of others. Behavioral economics explores the way in which our irrational behavior affects economies, and the results are curious and subtle. For example, the riddle of experience versus memory (TED video), or the several examples in The Marketplace of Perception (Harvard Magazine article). I think it would be illuminating to examine free software through this lens, and consider that the vagaries of human perception may have a very strong influence on our choices. Some questions for thought: If you re aware of any studies along these lines, I would be interested to read about them.

Russell Coker: Links May 2010

AdRevenge is an interesting concept to pay for Google Adsense adverts about how companies suck [1]. If a suitably large group of people pay to warn you about a company then it s a good signal that the company is actually doing the wrong thing. A guest post by Mili on Charles Stross blog has an interesting analysis of the economcs of Intellectual Property and concludes that content is a public good [2]. New Age Terrorists Develop Homeopathic Bomb [3], an amusing satire of medical fraud and security theatre. The sit has a lot of other good satire too. Mark Shuttleworth wrote an interesting post about new window management changes that will soon go into Ubuntu [4]. He points out that the bottom status bar in applications is a throw-back to Windows 3.1 and notes that a large part of the incentive for removing it (and using the title-bar for the status) is the work on the Netbook version of Ubuntu. This is really ironic given that the resolution of current Netbooks is quite similar to that of desktop systems that were current when Windows 3.1 came out. Omar Ahmad gave an insightful TED talk about the benefit of using a pen and paper to send a letter to a politician [5]. Sebastian Wernicke gave an amusing and informative TED talk about how to give a good TED talk [6]. His talk gives some useful ideas for public speaking that are worth considering. Catherine Mohr gave a brief and interesting TED talk about how to build an energy efficient house with low embodied energy [7]. Her blog at www.301monroe.com has the details. Stephen Wolfram (of Mathematica fame) gave an interesting TED talk [8]. He covers a lot of interesting things that can be done with computers, primarily based on the Wolfram Alpha [9] platform which allows natural language queries of a large data set. He also talks about the search for a Theory of Everything. Esther Duflo gave an interesting TED talk about using social experiments to fight poverty [10]. She describes how scientific tests have been used to determine the effectiveness of various ways of preventing disease and encouraging education in developing countries. One example of the effectiveness of such research is the DeWormTheWorld.org project which was founded after it was discovered that treating intestinal worms was the most cost effective way of getting African children to spend longer at school. David L. Rosenhan wrote an interesting research paper On Being Sane In Insane Places about pseudo-patients admitted to psychiatric hospitals [11]. It seems that psychiatric staff were totally unable to recognise a sane person who was admitted even though other patients could do so. It also documents how psychiatric patients were treated as sub-human. One would hope that things had improved since 1973, but it seems likely that many modern psychiatric hospitals are as bad as was typical in 1973. It s also worth considering the issue of the treatment in society of people who have been diagnosed with a mental illness, it seems likely that the way people are treated in the community would have similar bad results to that which was documented for treatment in psychiatric hospitals even the sanest people will act strangely if treated in an insane manner! Also it seems to me that there could be potential for using a panel of patients assembled via the Delphi Method as part of the psychiatric assessment process as it has been demonstrated that patients can sometimes assess other patients more accurately than psychiatrists! Simon Sinek gave an inspiring TED talk about how great leaders inspire action [12]. Of course the ideas he describes don t just apply to great leaders, they should apply to ordinary people who just want to convince others to adopt their ideas. Stephen Collins write a good article summarising the main reasons why the proposed great firewall of Australia is a bad idea [13]. Lenore Skenazy who is famous for letting her 9yo son catch the metro alone during broad daylight on a pre-planned route home has created a web site about Free Range Kids [14]. She seems to be starting a movement to oppose Helicopter Parenting and has already written a book about her ideas for parenting. The incidence of crime has been steadily increasing, as has the ability of the police to apprehend criminals and recover abducted children. There s no reason for children to be prevented from doing most of the things that children did when I was young!

John Goerzen: Free-Range Kids: A Parental Rant

There s a lot of unreasonable panic out there these days. Parents that don t want to let their kids even walk to the mailbox because of an irrational fear of abduction. (Children are more likely to die falling out of bed than be abducted, and before anyone panics, that s also exceptionally rare.) A woman that let her 11-year-old travel unaccompanied on the NYC subway he d been riding all his life unleashed a huge controversy. And there s also this maddening tendency for some parents to be so worried about wounding their children s self-esteem that they never want to say they ve done something wrong, and be so positive to them. Recent studies, as well as common sense, suggest that both attitudes are rather harmful. Terah and I are believers in letting children have cheap mistakes . If I see Jacob running through the kitchen about to trip over a toy he left out, I probably won t say anything. Him crashing to the floor after running on it won t even hurt him enough that he ll cry, but it will probably be far more effective at teaching safety than 20 days worth of me saying no running in the kitchen. Of course, we ll provide comfort for him when he s sad and be there to support him, but also give him more and more freedom to succeed or fail as he grows. This starts even younger. When he first started feeding himself, he got into this habit of throwing food on the floor. We had some natural consequences as a result. If his supper landed on the floor, then he would be all done eating. (Note: Jacob was always on the heavy side at that age anyway.) He couldn t reach his supper on the floor, and we weren t going to keep bending over to pick it up for him. Fixed that problem pretty quickly at least until he learned that he could wait until he was pretty full before throwing his food down on the floor (at which point we added a brief time-out). Or how about our reaction when he has mistakes. When he breaks something, whether accidentally or on purpose, we don t give him some sort of affirmation like Oh, that s OK, we can fix it or good try; you ll do better next time. It is NOT OK to rip the pull-up flaps out of a book or to stomp on those peas you dropped on the floor, and there will be predictable and reasonable consequences for doing so. As a result, when he does a good job, it ll mean more. He, at 3, has started sounding out letters, knows his alphabet, and can count to about 15 in English and about 3 in German. And those are things we give him a good job about. Two of the best things we can do for our children are: 1) be honest about their successes and failures; and 2) give them room to succeed and room to fail, so they can learn from both while the consequences mean a skinned knee or broken toy instead of a broken rib or foreclosure on a mortgage. The Free Range Kids FAQ makes excellent reading, and discusses how many of today s parents grew up enjoying this kind of freedom, and for various reasons are too scared to allow their children that today. Rather sad, really, especially given that today s world is safer.

24 May 2010

Carl Chenet: Nagstamon 0.9.3 in sid

Nagstamon 0.9.3 just landed in Sid. Nagstamon is a Nagios status monitor for the desktop. It connects to multiple Nagios or Opsview monitoring servers and resides in systray or as a floating statusbar at the desktop showing a brief summary of critical, warning, unknown, unreachable and down hosts and services and pops up a detailed status overview when moving the mouse pointer over it. Connecting to displayed hosts and services is easily established by context menu via SSH, RDP and VNC. Users can be notified by sound. Hosts and Services can be filtered by category and regular expressions.

Nagstamon in the toolbar

Successfully tested Nagios versions include 2.x and 3.x. Opsview connectivity has been tested with Opsview 3.5.

Hosts details with Nagstamon

If you don t like having a Nagios webpage always running in your web browser, you ll appreciate Nagstamon.

Lior Kaplan: Taking the rust of my debian packaging skills

582784 It s been a long time since my last upload to the archive. Actually more than a year and a half. This weekend I tried to get back to packaging, hoping to catch up with some of the needed work. The beginning was very weird, forcing me to review the developer references, the policy and the new maintainer guide for information I already forgot. After a few hours of hacking, it felt like I wasn t MIA at all. After 72 hours, I quite happy with the results: Another issue I started and not sure how to continue is finding programs who lost their bidi support due to the last upgrade of libfribidi-dev. In the last version (0.19.x), the fribidi-config utility was dropped by upstream and pkg-config should be used instead. The problem is that programs which build against libfribidi-dev, look for fribidi-config and FTBFS (see #571351) or ignore it s disappearance and stop providing bidi support (see #582784).
Filed under: Debian GNU/Linux

Julien Danjou: Making startup-notification XCB native

I'm trying to work on XCB this week. And today I've started to accomplish the second step of a long term goal: making an X11 only library using XCB as its primary interface instead of Xlib. Last year, I had extended the API of startup-notification to support XCB as a back-end. This had been made possible by factorizing some code, duplicating the X11 code and translating it into equivalent XCB. Today, I've accomplished the second step, being dropping the Xlib code inside startup-notification to keep only the XCB one. For this, I used the x11-xcb library, which is available when Xlib is compiled with XCB as its transport, which is nowadays the standard. This library provides the function XGetXCBConnection, which can convert a Display pointer to a xcb_connection_t pointer. Consequently, it's now possible to write and execute XCB based code and being compatible with Xlib. I've made some benchmark of my work for the occasion, in order to measure what the gain is. The first table described 1000 launches of a fake application (a modified version of the startup-notification test suite actually). The X server is local (the latency is very minimal then). The gain is computed between the same back-end type for the total time. Full XCB is the "version" I'm working on.
Version - Back-end User time (seconds) Kernel time (seconds) Total time (seconds) Gain
0.10 - libx11 3.20 7.42 12.989 -
0.10 - libxcb 2.76 7.36 12.414 -
Full XCB - libx11 2.74 7.50 12.380 4.6 %
Full XCB - libxcb 2.72 7.16 12.037 3.0 %
The user time and kernel time are provided but are not really interesting. XCB does not offers a big gain in CPU execution time, but is more about latency. Anyhow, there's always a gain with XCB. This second table describe the same test but running only 100 times over a slow network.
Version - Back-end Total time (seconds) Gain
0.10 - libx11 76 -
0.10 - libxcb 35 -
Full XCB - libx11 72 5.2 %
Full XCB - libxcb 33 5.7%
The gain is relatively small, about 5 %. But anyhow, there's still a gain. Note that the difference between the execution time of the same test written in XCB and Xlib is just huge. I've tried to optimize the Xlib test, but I did not manage to win more seconds. In conclusion, considering that startup-notification is only used when an application launches another application, the perceivable gain might be even smaller. But anyhow, I think it's worth it.

Axel Beckert: Perfect Team: autossh and GNU Screen

SSH is definitely one of my most often used tools, not only for system administration at work but also on the road with my netbook, an EeePC 701 running Debian Sid. On the road, it often happens that I have a flaky WLAN or UMTS connection, so I often have to kill (via <Enter>~.) and reconnect my SSH session due to a changed IP address or so. First step against problems arising from using SSH over unreliable network connections is of course GNU Screen. Second step is use SSH keys and ssh-agent to not needing to type the password on every reconnect. But it s still very annoying to kill the SSH connection and call ssh again manually. For luck there is autossh, a wrapper around SSH which regularily checks via two tunnels connect to each other on the remote side if the connection is still alive, and if not, it kills the ssh and starts a new one with the same parameters (i.e. tunnels, port forwardings, commands to call, etc.). It s quite obvious that this is perfect to be combined with screen s -R and -d options (Reattach if a detached screen is around, else start a new screen; detach a currently attached screen), so I found myself very often typing (or fetching it from the commandline history :-):
autossh -t sym.noone.org 'screen -Rd'
-t is necessary to allocate a terminal device on the remote machine which is not done by default if you directly call a command via ssh. In comparision to OpenSSH, autossh needs the single quotes, because otherwise it would parse -Rd as options to parse to ssh and bail out. That s not a real problem, but when you re used to just type ssh -t sym.noone.org screen -Rd without any quotes, you ll run into this then and when. Update, 25-May-2010, 14:55: As Carsten Hey points out, autossh also supports the -- option to declare that all following options and parameters must be passed to ssh itself. (End of Update) Typing that often and mistyping it then and when cries for an shell alias or an shell function. So I came up with the following shell function:
asc()  
    autossh -x -a -t "$@" 'screen -RdU'
 
I used a function instead of an alias in case of autossh will in future regard all parameters given after the command as part of the command as ssh does. The additional options -x and -a disable X and SSH Agent forwarding which both don t work if you reattach to an already running screen. And if you re using Zsh as I do, you can even add some more format string magic to set the window title more or less to the expanded alias, eh, function:
function asc()  
    # Set the title to something more obvious, e.g. the expanded
    # alias, eh, function
    print -Pn "\e]0;%n@%m: autossh -t $* 'screen -RdU'\a";
    autossh -x -a -t "$@" 'screen -RdU'
 
compdef asc=ssh
Update, 25-May-2010, 14:59: As Hauke points out in a comment, Zsh users should also declare that asc should have the same tab completion as ssh itself. The example above has been updated accordingly. (End of Update) In the meanwhile on the EeePC I use asc on the commandline more often than ssh itself. And I nearly no more type autossh. (The most common exception here is autossh hostname tail -F /path/to/some/logfile.) Using that function you can also add common ssh options for tunneling, etc. I use it most often like this:
asc -D 1080 sym.noone.org
This opens a SOCKS proxy on localhost, port 1080 and that way I can surf via the host I m connecting to by SSH. There s one small drawback though: You didn t expect that I can just invent some new three letter command without a namespace clash, did you? There is a free game called Advanced Strategic Command whose binary (and Debian package) is named asc, too. If you have that game installed, you can always call it using its full path, e.g. /usr/games/asc on Debian. P.S.: My whole grml based .zshrc is also available via git at git.noone.org as well as on github.

23 May 2010

Marco T&uacute;lio Gontijo e Silva: Playing with criterion

As a part of my Google Summer of Code project, I should take a look at criterion, a benchmark library for Haskell. As I d have to use it, and it seems to be a useful library for a lot of people, I packaged it to Debian, with all its dependencies. The haskell-criterion package is waiting on the NEW queue. Ok, so now I had to test it with something. I came up with a test related to my reading about the Haskell execution model. I noticed that I never have played with multi-threaded Haskell code to achieve better performance, but only to make things that should be in parallel work. So I thought this was a good oportunity to try both. I wrote this code: import Control.Applicative ((<$>), (<*>))
import Control.Concurrent.MVar (newEmptyMVar, putMVar, takeMVar)
import Control.Exception (evaluate)
import Criterion.Main (Benchmark, bench, bgroup, defaultMain, whnf, whnfIO)
import GHC.Conc (forkIO, par)
import System.Environment (getArgs) value :: Int
value = 32 threadsL :: [Int]
threadsL = [1, 2, 3, 4, 8, 16, 32, 64] main :: IO ()
main
= defaultMain
[bgroup "fib" $ [bench "fib" $ whnf fib value]
, bgroup "pfib" $ map pbench threadsL
, bgroup "ifib" $ map ibench threadsL] pbench :: Int -> Benchmark
pbench i = bench (show i) $ whnf (pfib i) value ibench :: Int -> Benchmark
ibench i = bench (show i) $ whnfIO $ ifib i value fib :: Int -> Int
fib n
n < 2 = n
otherwise = fib (n 1) + fib (n 2) pfib :: Int -> Int -> Int
pfib threads n
n < 2 = n
threads < 2 = fib n
otherwise = par n1 $ seq n2 $ n1 + n2
where
n1 = pfib (threads 1) $ n 1
n2 = fib $ n 2 ifib :: Int -> Int -> IO Int
ifib threads n
n < 2 = evaluate n
threads < 2 = evaluate $ fib n
otherwise
= do
f1 <- newEmptyMVar
f2 <- newEmptyMVar
forkIO $ ifib (threads 1) (n 1) >>= putMVar f1
(+) <$> evaluate (fib $ n 2) <*> takeMVar f1 My point is to benchmark the na ve implementation of the Fibonacci function. I ve done three implementations:
fib
which is the simple sequential algorithm
pfib
which call par threads times
ifib
which call forkIO trheads times
I have chosen the value to be found to 32, because it took long enought to make the algorithms show a difference, and short enough to allow me to wait for the benchmarks. I built with:
ghc --make -threaded fib
And runned with:
./fib --summary=fib.csv +RTS -N
The results were surprising for me:
Name,Mean,MeanLB,MeanUB,Stddev,StddevLB,StddevUB
"fib/fib",1.7197746428132055,1.5104790505051613,2.031735854971409,1.2921598378638277,0.9514102096387,1.7188536150257019
"pfib/1",1.246479478704929,1.1805571945786482,1.3496261318802836,0.41570319371618925,0.29446316107198395,0.5823241217769125
"pfib/2",0.709402781355381,0.6850755818963056,0.7553855475068096,0.1650893842889147,0.10136189180677138,0.26853312094492815
"pfib/3",0.6876571162819864,0.6694172366738321,0.7332831247925761,0.136836636420341,5.5142145935712775e-2,0.2581805596992922
"pfib/4",0.6940202649712561,0.6684220250725745,0.746035416948795,0.17779646229098522,9.15862482237111e-2,0.2804203635281818
"pfib/8",0.5887631162285805,0.5625207790017128,0.637939236986637,0.17967127569396815,0.10516397607491562,0.2680488710546616
"pfib/16",0.6578513725876809,0.6107277926087382,0.7372924216866497,0.3070809683902726,0.20679305873839168,0.44571891526087154
"pfib/32",0.8900135167717932,0.8123333438515664,0.9942556460976597,0.45942103747727,0.36818132854060703,0.6233813540278427
"pfib/64",0.6894888337731359,0.6386897596001626,0.7766118057847021,0.33329115700425754,0.22490082647959764,0.5323965317477072
"ifib/1",1.3290104445099833,1.2501559790253634,1.4700443919777875,0.5277373265779313,0.35208278898795436,0.9179652474421286
"ifib/2",0.7626163347840308,0.7239342554688452,0.8381309326767921,0.26785654248385327,0.1679981089775395,0.44853454282169086
"ifib/3",0.7440206035256386,0.6907905062317847,0.8396386941552162,0.3549675945489581,0.22865300516283724,0.536883746220313
"ifib/4",0.6503480085015301,0.6130640967965126,0.711405030119419,0.239522235814317,0.1712396398224678,0.3616636557596486
"ifib/8",0.6300301631569865,0.6085795434594155,0.699028070795536,0.17776228649867923,6.103918021418795e-2,0.39508078831864085
"ifib/16",0.6735970553040505,0.6393461784005164,0.7405966480851172,0.2366561170051713,0.13725890735483465,0.38273830961617655
"ifib/32",0.6162510784745214,0.5983195599198344,0.6736035832047466,0.14678572822644564,5.335959472427259e-2,0.3231734735693349
"ifib/64",0.7217859562516212,0.6820346101403237,0.7857911285042766,0.2534368199920431,0.18073576275895306,0.37636526159654315
I m only analyzing the first column. How come pfib/1 and ifib/1, that is, pfib and ifib called with just one thread, got faster than fib/fib? All they do is calling fib! There s certainly something wrong here. So I reran the tests in a single user environment (recovery-mode on GRUB). And also learned that I should not trust blindly these results in my usual user environment. The new tests were also surprising:
Name,Mean,MeanLB,MeanUB,Stddev,StddevLB,StddevUB
"fib/fib",0.4979443445852824,0.4978961411169597,0.49802434593949996,3.1186229071682967e-4,2.1763919616793833e-4,4.8569259617552764e-4
"pfib/1",0.4980979338339395,0.4980578795126507,0.49816361099992496,2.58930038115918e-4,1.7011790805380493e-4,4.0161943492611696e-4
"pfib/2",0.30187882572923386,0.3018276229551859,0.3019332328489849,2.7088498930449747e-4,2.3482715228451473e-4,3.21989842698397e-4
"pfib/3",0.3025411692312786,0.3024852743795938,0.3026018276861736,2.975260476849551e-4,2.6907713057828055e-4,3.3200184242397705e-4
"pfib/4",0.3045588889769145,0.3024707928350993,0.31463312775407504,2.0236423610294995e-2,3.5108499279216634e-4,4.8255009076253054e-2
"pfib/8",0.27702674538408,0.27158569247041425,0.2846977105787821,3.2790225595837955e-2,2.5215205278321072e-2,4.6554023411818295e-2
"pfib/16",0.26909642369065967,0.2650135436705178,0.2758063927343913,2.615724515115467e-2,1.8039283932648264e-2,4.0956608339390285e-2
"pfib/32",0.2726816359213419,0.26952718407426557,0.27636309534822184,1.7444576031246224e-2,1.4227057669027803e-2,2.2935079362453085e-2
"pfib/64",0.28010295302186694,0.27646669537339885,0.28658743054185587,2.4301648337333875e-2,1.3360828345387364e-2,3.9546300072208655e-2
"ifib/1",0.4984497705153056,0.4984218493155071,0.4984962406805583,1.8040276986757106e-4,1.242720342516858e-4,2.870577387900989e-4
"ifib/2",0.3603615990332194,0.34868544012818997,0.3741170039824078,6.496311767153333e-2,5.652406802832432e-2,7.448312251646773e-2
"ifib/3",0.31676718384538366,0.3109451881102153,0.32481522709642136,3.4887028192202074e-2,2.744363202200484e-2,4.8060797483303504e-2
"ifib/4",0.31541500956330976,0.30635046393190096,0.32643843323503213,5.109215651574664e-2,4.339074296796736e-2,5.995395740457435e-2
"ifib/8",0.32055945069108693,0.3132960501364299,0.3302640953711101,4.2886292109838166e-2,3.422708481199234e-2,5.4196317284934535e-2
"ifib/16",0.32268204123292654,0.31607536703859057,0.33141186625276287,3.858016286486453e-2,3.0959817731033304e-2,4.801572113400926e-2
"ifib/32",0.32020341069017116,0.3151469627073833,0.32664997250352584,2.92266723352992e-2,2.3352577028246314e-2,3.5913110786596904e-2
"ifib/64",0.3183670273474284,0.3136345234564373,0.3248856058767864,2.834147907711714e-2,2.1803978149062524e-2,3.8088053144371074e-2
I didn t thought the improvement in performance on a single-user would be so dramatic. The execution of the program in a single-user environment used 41% of the time to run in my usual environment. About the comparative results, they were more close to what I d expect than the old ones. fib/fib, pfib/1 and ifib/1 had a very similar execution time. What surprised me was that I was expecting to have the best result in the benchmarks with 2 threads, since I ran the tests in a Core 2 Duo T5750 @ 2.00 GHz with 2 cores. But in pfib the execution time has reduced from 4 threads to 8, and in ifib, from 2 to 3. I can t think of where these improvements come from. Unfortunately the creation of charts is currently broken in GHC 6.12, so no graphics by now.

22 May 2010

Ben Hutchings: Cancelling a command in bash

Rog rio Brito writes:
[...] whenever I was typing a command line, changed my mind and pressed C-c, I got a ^C printed on the screen, usually overwriting one or two characters of what I had typed. And this prevented me from automatically copying and pasting the command that I had typed.
There is an alternative to using C-c, and that is M-#. This adds a '#' to the start of the line, commenting it out, and then behaves as if you pressed Return. The result is that the command is cancelled but still remains on-screen and in your history. You can then copy the command later using the mouse or keyboard.

Rog rio Brito: Changing Some Configuration from Defaults

Two of my most frequently used tools are, hands down, Emacs and bash. Having a pleasant configuration of one s toolbox is necessary for long-term productivity. However, newer versions of the programs (any programs) don t always have good, comfortable defaults. That is the case with Emacs 23 and bash 4. With Emacs 23, by default, the scratch buffer shows some text that, if you want to quickly enter text to be saved, has to be deleted. This can be tolerated for some invocations of the editor. But at some point, it gets on one s nerves. Luckily, Emacs is so fully customizable that this is not a problem, unless you have to hunt down the right knobs and you are an e-lisp illiterate. That is my case. After reading an admittedly small amount of documentation, some good lines that ought to be put into one s .emacs file are:
(setq inhibit-startup-echo-area-message "rbrito")
(setq initial-scratch-message nil); show an empty scratch buffer
(setq inhibit-startup-message t); remove splash screen
Another line that may make things more comfortable (especially if you, like me, use Emacs s dired as a filemanager to navigate your folders):
(defalias 'yes-or-no-p 'y-or-n-p)
But configuring Emacs is not always that hard. Seeing a change of behaviour with your shell is scarier. When bash 4 was released, one thing that got implemented is the echo of some characters from the terminal to the screen. In practice and with my usage pattern, this meant that whenever I was typing a command line, changed my mind and pressed C-c, I got a ^C printed on the screen, usually overwriting one or two characters of what I had typed. And this prevented me from automatically copying and pasting the command that I had typed. Looking for an answer to that was much harder than to see how to change a parameter on Emacs: every search that I tried on Google with the words bash and Ctrl-C or some variants of that eventually lead me to how do I trap a Ctrl-C on my shell script? . Decided to see what changed, I read the documentation of readline. It didn t help, because I was reading something from version 5. I, then, downloaded the source code and started looking for SIGINT. Then, perusing the code (and the changelog), I saw that with bash 4 and readline 6, a new configuration option was introduced: the echo-control-characters option of readline. Of course, after knowing this, stopping the undesired behaviour was a simple matter of inserting set echo-control-characters off in my .inputrc and think about how much time I spent with that uncomfortable behaviour in some of my preferred tools. As an addendum to the previous paragraphs (written on February of this year, but only now published), the Debian package of Emacs 23.2, the newest version as of this writing, comes with some configuration that makes it ignore the settings that are used with fontconfig. I did not spend any time trying to see the cause of the problem, but adding a simple
Emacs*font:			Mono
to my .Xdefaults file did the trick and restored back the ability to use DejaVu Sans Mono at the right size on my system.

Petter Reinholdtsen: More flexible firmware handling in debian-installer

After a long break from debian-installer development, I finally found time today to return to the project. Having to spend less time working dependency based boot in debian, as it is almost complete now, definitely helped freeing some time. A while back, I ran into a problem while working on Debian Edu. We include some firmware packages on the Debian Edu CDs, those needed to get disk and network controllers working. Without having these firmware packages available during installation, it is impossible to install Debian Edu on the given machine, and because our target group are non-technical people, asking them to provide firmware packages on an external medium is a support pain. Initially, I expected it to be enough to include the firmware packages on the CD to get debian-installer to find and use them. This proved to be wrong. Next, I hoped it was enough to symlink the relevant firmware packages to some useful location on the CD (tried /cdrom/ and /cdrom/firmware/). This also proved to not work, and at this point I found time to look at the debian-installer code to figure out what was going to work. The firmware loading code is in the hw-detect package, and a closer look revealed that it would only look for firmware packages outside the installation media, so the CD was never checked for firmware packages. It would only check USB sticks, floppies and other "external" media devices. Today I changed it to also look in the /cdrom/firmware/ directory on the mounted CD or DVD, which should solve the problem I ran into with Debian edu. I also changed it to look in /firmware/, to make sure the installer also find firmware provided in the initrd when booting the installer via PXE, to allow us to provide the same feature in the PXE setup included in Debian Edu. To make sure firmware deb packages with a license questions are not activated without asking if the license is accepted, I extended hw-detect to look for preinst scripts in the firmware packages, and run these before activating the firmware during installation. The license question is asked using debconf in the preinst, so this should solve the issue for the firmware packages I have looked at so far. If you want to discuss the details of these features, please contact us on debian-boot@lists.debian.org.

Wouter Verhelst: Help!

Yes, there is something like an 'RFH', or 'Request For Help', in the WNPP in Debian, but I thought I'd go through this blog, first, since I believe many more people who will feel inclined to help here will read this blog. I've been maintaining the beid packages in Debian since, eh, quite a while now; and while it's not always been the package that I have spent the most time on, it's not a package that I want to give up. However, in recent times, I've had issues with the code. I've not really been able to get the 3.5.2 release to compile properly as a package, which means that for the moment, squeeze is still stuck with a 2.x release. This is a problem, since that old release isn't supported anymore upstream, and what's worse, it does not support newer-issued eID cards. So if your card is less than about two or three years old, you can't even use it anymore. For this reason, the beid packages really, really should be updated before squeeze releases. I've been trying to spend some time doing so, but real life kept intervening. And I haven't even been able to start looking for the 3.5.3 code that I'd heard rumours would exist. So, this is a request for help. If you want a more recent beid package to appear in Debian (and, by extension, Ubuntu) any time soon, and you can spare some extra time on maintaining a somewhat convoluted package, you'd be an ideal candidate for comaintainer of this package. You don't need to have any experience working on Debian, but you do need experience with build systems and Makefiles. If you don't have any experience in working on Debian, but would like to learn, this might be an interesting opportunity. If you do have experience working on Debian and are as interested as me in getting this to be up-to-date in the next Debian release, your help would be very much appreciated. Even if you only have time for a little while and would like me to take over once everything is going fine again, that's helpful. At any rate, if you want to help, contact me at wouter@debian.org. We'll work out the specifics then. Thanks in advance,

Russell Coker: A School IP Project

The music industry seems fairly aggressive in taking legal action against children when they break the licence terms of copyright material. I think it would be good to teach children about how the IP industry really works. It seems to me that you could have a school project that involves an entire year level (maybe 100 students depending on the size of the school) each of whom can produce copyright material (everything they do in art and English classes would be suitable as a start). Then they could register their work (make digital photographs and then store them in a school database that records the entry date) and sue anyone who infringes their work. Every student would receive licence fees for their work, but if they are sued for infringement they would have to pay all revenue plus damages. Other students could work as lawyers and take a portion of the proceeds of any successful law suit, and finally some students could run recording companies and spend their time hunting for infringing work for the purpose of launching legal action. In terms of the licence fees paid, this could be done by just allocating a fixed value per item to each student as a way to get the system running without regard to the fact that some students just aren t able to create good art. It could however have a large portion of the value coming from what other students choose to spend, every student gets to spend $10 per week on art and they can choose from the database what they want to buy copies of. The most popular art could then be printed on every notice-board in the school as an incentive for students to vote with their play-money for something that they don t mind seeing all the time. It s obvious that popularity would be a significant factor in the success of some artists, but that s OK, a casual review of the chart topping music reveals that it s quite obviously not created by the world s best musicians so it seems that rewarding popularity rather than skill just adds some realism. One possibility would be to allow the students to elect representatives to create their own IP laws. It would be interesting to see how the IP laws voted on by representatives of the students (who are all in some way involved with the process of creating, buying, selling, and distributing artistic products) differ from those which we have foisted upon us in the real-world. Also an interesting possibility would be to allow corruption in the election process and observe how the results differ from year levels where corruption is not permitted. I expect that teaching children how political corruption works would be a little controversial, but it s nothing that they can t learn from reading news reports about what the entertainment industry is really doing. Really being a corrupt politician for a school project shouldn t be as bad as playing a murderer in a school play! Naturally this couldn t be done with real money, but giving higher marks at the end of the year to the students who accumulate the most play money would be quite reasonable. I don t think that there would be a problem with giving higher marks to a student who succeeded through political corruption as long as they gave a good written report of how they did so and the implications for society. Please note that I am not suggesting this for a subject that is used for university entrance, I think it would be a good project for years 8-10 which in Australia have no relevance to university entrance. So the marks would just be letters on a bit of paper that might make parents happy or unhappy and otherwise mean nothing. I anticipate responses from people who believe that educating children about how the world works is not appropriate for a school. Such people are never going to convince me, but if anyone thinks that they can make a good point to convince some of the readers then I encourage them to write it up in the comments section if it s short or on their own blog if it s longer.

Next.

Previous.