Search Results: "ana"

26 June 2012

Ingo Juergensmann: Confusion about mkfs.xfs and log stripe size being too big

Recently I bought some new disks, placed them into my computer, and built a RAID5 on these 3x 4 TB disks. Creating a physical device (PV) with pvcreate, a volume group (VG) with vgcreate and some logical volumes (LV) with lvcreate was as easy and well-known as creating an XFS filesystem on the LVs... but something was strange! I never saw this message before, when creating XFS filesystems with mkfs.xfs:
log stripe unit (524288 bytes) is too large (maximum is 256KiB)
log stripe unit adjusted to 32KiB
Usually I don't mess around with the parameters of mkfs.xfs, because mkfs.xfs is smart enough to find near to optimal parameters for your filesystem. But apparently mkfs.xfs wanted to use a log stripe unit of 512 kiB, although its maximum size for this is 256 kiB. Why? So I started to google and in parallel asked on #xfs@freenode. Erik Sandeen, one of the core developers of XFS, suggested that I write that issue to the mailing list. He did already face this issue himself, but couldn't remember details. So I collected some more information about my setup and wrote to the XFS ML. Of course I included information about my RAID5 setup:
muaddib:/home/ij# mdadm --detail /dev/md7
/dev/md7:
Version : 1.2
Creation Time : Sun Jun 24 14:58:21 2012
Raid Level : raid5
Array Size : 7811261440 (7449.40 GiB 7998.73 GB)
Used Dev Size : 3905630720 (3724.70 GiB 3999.37 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent Update Time : Tue Jun 26 05:13:03 2012
State : active, resyncing
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0 Layout : left-symmetric
Chunk Size : 512K Resync Status : 98% complete Name : muaddib:7 (local to host muaddib)
UUID : b56a714c:d193231e:365e6297:2ca61b65
Events : 16 Number Major Minor RaidDevice State
0 8 52 0 active sync /dev/sdd4
1 8 68 1 active sync /dev/sde4
2 8 84 2 active sync /dev/sdf4
Apparently, mkfs.xfs takes the chunk size of the RAID5 and want to use this for its log stripe size setting. So, that's the explanation why mkfs.xfs wants to use 512 kiB, but why is the chunk size 512 kiB at all? I didn't messed around with chunk sizes when creating the RAID5 either and all of my other RAIDs are using chunk sizes of 64 kiB. The reason was quickly found: the new RAID5 has a 1.2 format superblock, whereas the older ones do have a 0.90 format superblock. So, it seems that somewhen the default setting in mdadm, which superblock format is to be used for its metadata, has been changed. I asked on #debian.de@ircnet and someone answered that this was changed in Debian after release of Squeeze. Even in Squeeze the 0.90 format superblock was obsolete and has been only kept for backward compatibility. Well, ok. There actually was a change of defaults, which explains the behaviour of mkfs.xfs now, wanting to set log stripe size to 512 kiB. But what is the impact of falling back to 32 kiB log stripe size? Dave Chinner, another XFS developer explains:
Best thing in general is to align all log writes to the
underlying stripe unit of the array. That way as multiple frequent
log writes occur, it is guaranteed to form full stripe writes and
basically have no RMW overhead. 32k is chosen by default because
that's the default log buffer size and hence the typical size of
log writes.

If you increase the log stripe unit, you also increase the minimum
log buffer size that the filesystem supports. The filesystem can
support up to 256k log buffers, and hence the limit on maximum log
stripe alignment.
And in another mail, when being asked if it's possible to raise the 256 kiB limit to 512 kiB because of the mdadm defaults to 512 kiB as well:
You can't, simple as that. The maximum supported is 256k. As it is,
a default chunk size of 512k is probably harmful to most workloads -
large chunk sizes mean that just about every write will trigger a
RMW cycle in the RAID because it is pretty much impossible to issue
full stripe writes. Writeback doesn't do any alignment of IO (the
generic page cache writeback path is the problem here), so we will
lamost always be doing unaligned IO to the RAID, and there will be
little opportunity for sequential IOs to merge and form full stripe
writes (24 disks @ 512k each on RAID6 is a 11MB full stripe write).

IOWs, every time you do a small isolated write, the MD RAID volume
will do a RMW cycle, reading 11MB and writing 12MB of data to disk.
Given that most workloads are not doing lots and lots of large
sequential writes this is, IMO, a pretty bad default given typical
RAID5/6 volume configurations we see....
So, reducing the log stripe size is in fact a good thing[TM]. If anyone will benefit from larger log stripe sizes, s/he would be knowledgeable enough to play around with mkfs.xfs parameters and tune them to needs of the workload. Erik Sandeen suggested, though, to remove the warning in mkfs.xfs. Dave objects and maybe it's a good compromise to extend the warning by giving an URL for a FAQ entry explaining this issue in more depth than a short warning can do? Maybe someone else is facing the same issue and searches for information and find this blog entry helpful in the meantime...
Kategorie:

Russ Allbery: Review: Quiet

Review: Quiet, by Susan Cain
Publisher: Crown
Copyright: 2012
ISBN: 0-307-45220-4
Format: Kindle
Pages: 263
I've always been an introvert. This is something that surprises some people when they first meet me since they equate introversion with shyness, and I'm not at all shy. It surprises others because I'm quite vocal and outspoken in meetings, but introversion also doesn't imply a lack of self-confidence. I can run meetings, give presentations, and argue my perspective in front of just about anyone, but I don't like parties, I crave time by myself, and I could happily go for weeks without seeing another human being. I'm an introvert because I find people draining rather than invigorating, written communication far easier and more comfortable than spoken, and superficial social contact more irritating and frustrating than enjoyable. If you think think that means there may be something wrong with me, or that I would be happier if "drawn out of my shell," I wish you would read this book. But I suspect its core audience will be people like me: those who are tired of being pushed to conform with extrovert beliefs about social interaction, those who are deeply disgusted by the word "antisocial" or feel pangs of irrational guilt when hearing it, or those who just want to read an examination of interpersonal interactions that, for once, is written by and about people who like quiet and solitude just like they do. I first encountered Susan Cain via her TED talk, which I think is both the best possible summary of this book and the best advertisement for it. If you've not already seen it, watch it; it's one of the best TED talks I've seen, good enough that I've watched it three times. If you then want more of the same, buy Quiet. Quiet has, I think, three messages. First, it's a tour of the science: what is introversion and extroversion? Is there evidence that these are real physiological differences? (Spoiler: yes.) What do we know about introversion? How do we know those things what experiments have been done and what methods have been used? Here, it's a good general introduction, although Cain is careful to point out that it only scratches the surface and there's much more scientific depth. For example, she touches on the connections between introversion and sensitivity to stimulus and points out that they're two separate, if related, categorizations, but doesn't have the space here to clarify the distinctions and tease them apart. But she lays a reasonable foundation, particularly in defense of introversion as a natural, physiologically grounded, scientifically analyzable, common, and healthy way of interacting with the world. (For those who are curious about the distinctions between introversion and sensitivity, and the argument that most of what Cain says here about introversion is actually about sensitivity, see the blog post by Elaine Aron.) The second message, the one that resonated with me the most, was Cain's passionate defense of introversion. Business culture (at least in the United States, which is what both Cain and I know) is strongly biased towards extroversion; at least faking extroversion seems to be required for some career advancement. Extrovert culture dominates politics and most public discourse. It's common to find people who consider introversion, particularly in children, to be a sign of unhappiness, poor social adjustment, psychological problems, or other issues that should be "cured" or changed. Cain's gentle but firm passion in defense of introversion is a breath of fresh air. She attacks open plan offices, the current obsession with group learning and social school settings, and the modern group-think bias towards collaboration over solitude and concentration, and she does that with a combination of polite frustration and the conclusions of multiple studies. Introverts will be cheering as she constructs solid arguments and musters evidence against things that we've always found miserable and then been told we were wrong, short-sighted, or socially inept for finding miserable. I am so utterly on her side in this argument that I have no way of knowing how persuasive it will be, but it's lovely just to hear someone put into words what I feel. This defense does skew the book. Quiet is not, and does not purport to be, an even-handed presentation of introversion and extroversion. It's written proudly and unabashedly from the introvert's point of view. I'm fine with that: I, like Cain, think the US is saturated in extrovert perspectives and extrovert advice, particularly in the business world, and could use some balancing by activism from the other perspective. But be aware that this is not the book to look to for an objective study of all angles of the introvert/extrovert dichotomy, and I'm not sure her descriptions of extroversion are entirely fair or analogous to those of introversion. The extroversion described here seems somewhat extreme to me. I'm dubious how many extroverts would recognize themselves in it, which partly undermines the argument. The third message of the book, once Cain has won the introvert's heart, is some advice on how to be a proud introvert, to make the space and find the quiet that one desires, and to balance that against places where one may want and need to act like an extrovert for a while. Cain thankfully does not try to make this too much of the book, nor does she hold up any particular approach as The Answer. All the answers are going to be individual. But she does offer some food for thought, particularly around how to be conscious of and make delibrate choices about one's energy expenditures and one's recharge space. She also captures beautifully something that I've not seen explained this well before: the relief that an introvert can feel in the company of an extrovert who helps navigate social situations, make conversation, and keep discussions going until they can reach the depth and comfort level where the introvert can engage. I wish anyone in a position of authority over social situations would read this book, or at least watch the TED talk and be aware of the issues. Particularly managers, since (at least in my relatively limited experience) workplace culture is so far skewed towards extroversion that it can be toxic to introverts. Many of the techniques used by extrovert managers, and the goals and advice they give their employees, are simply wrong for introverts, and even damaging. Cain speaks well to the difficulties of empathy between people with very different interaction preferences, such as the problems with extroverts trying to "draw out" introverts who have hit social overload (or sensitive people who have hit stimulus overload). She also discusses something that I'd previously not thought about, namely how the pressure towards extroversion leads people to act extroverted even when naturally introverted, and how it's therefore very difficult to tell from behavior (and sometimes even to tell internally!) what one's natural interaction style is. But mostly I recommend this book if you're an introvert, if the TED talk linked above speaks to you. Even if we can't convince the world to respect introversion more, or at least stop treating it as abnormal, it's a lovely feeling to read a book from someone who gets it. Who understands. Who fills a book with great stories about introverts and how they construct their worlds and create quiet space in which to be themselves. Rating: 9 out of 10

24 June 2012

Pablo Lorenzzoni: Zotero and note-taking

I was looking for an excuse to try Zotero and the perfect opportunity appeared when I got a whole lot of references to group for a month of Magnetic Ressonance studies I am currently taking. I was also pleased to notice it is packaged to Debian. I am used to note-taking software. Back when I used a Palm m130 (and a Treo 650), I managed a lot of Memos I eventually migrated to Note-Everything in my current Android phone. Zotero, unfortunatelly, is not interfaceable with my phone (or I was still unable to figure out how to do so), but it s powerful in managing references beyond simple note-taking. Is anyone using Zotero in a more ambitious way? I ve read about people using it to keep large researchs to support fiction and non-fiction book-writing I also watched some YouTube videos on it. As far as I went with it, Zotero might become an important piece in my toolbox wrt reference keeping, so I was just trying to figure out how many other niches it can fill

Joey Hess: trying obnam

Obnam 1.0 was released during several months when I had no well-connected server to use for backups. Yesterday I installed a terabyte disk in a basement with a fiber optic network connection, so my backupless time is over. Now, granted, I have a very multi-layered approach to backups; all my data is stored in git, most of it with dozens of copies automatically maintained, and with archival data managed by git-annex. But I still like to have a "real" backup system underneath, to catch anything else. And to back up those parts of my user's data that I have not given them tools to put into git yet... My backup server is not my basement, so I need to securely encrypt the backups stored there. Encrypting your offsite backups is such a good idea that I've always been surprised at the paucity of tools to do it. I got by with duplicity for years, but it's increasingly creaky, and the few times I've needed to restore, it's been a terrific pain. So I'm excited to be trying Obnam today. So far I quite like it. The only real problem is that it can be slow, when there's a transatlantic link between the client and the server. Each file backed up requires several TCP round-trips, and the latency kills the bandwidth. Large files are still sent fast, and obnam uses little resources on either the client or server while running. And this mostly only affects the initial, full backup. But the encryption and ease of use more than make up for this. The real killer feature with Obnam's encryption isn't that it's industry-standard encryption with gpg, that can be trivially enabled with a single option (--encrypt-with=DEADBEEF). No, the great thing about it is its key management. I generate a new gpg key for each system I back up. This prevents systems reading each other's backups. But that means you have to backup the backup keys.. or when a system is lost, the backup would be inaccessible. With Obnam, I can instead just grant my personal gpg key access to the repository: obnam add-key --keyid 2512E3C7. Now both the machine's key and my gpg key can access the data. Great system; can't revoke access, but otherwise perfect. I liked this so much I stole the design and used it in git-annex too. :) I'm also pleased I can lock down .ssh/authorized_keys on my backup server, to prevent clients running arbitrary commands. Duplicity runs ad-hoc commands over ssh, which defeated me from ever locking it down. Obnam can be easily locked down, like this: command="/usr/lib/openssh/sftp-server" This could still be improved, since clients can still read the whole filesystem with sftp. I'd like to have something like git-annex's git-annex-shell, which can limit access to only a specific repository. Hmm, if Obnam had its own server-side program like this, it could stream backup data to it using a protocol that avoids the roundtrips needed by the SFTP protocol, and fix the latency issue too. Lars, I know you've been looking for a Haskell starter project ... perhaps this is it? :)

23 June 2012

Bernd Zeimetz: Report from the Bug Squashing Party in Salzburg

bsp_2012_salzburg photo from salzburg-cityguide.com, Copyright (C) Uwe Brandl

Participation and Results From June 15-17th we held a Debian BugSquashingParty in Salzburg, hosted and sponsored by conova communications GmbH. It was a fun and busy weekend, with 15-17 people from 5 countries being around, mainly working on RC bugs in Testing/Unstable. Gerfried Fuchs (rhonda) also worked on triaging the impact of RC bugs on the version in Squeeze, while Peter Palfrader (weasel) took care of Tor related things and Debian sysadmin work, including starting on the new bugs and udd hosts. Phillip Hug (hug) worked on the debian.ch infrastructure. Together with Miroslav Such from Red Hat Bernd Zeimetz (bzed) worked on the packaging of the necessary libraries and daemons to add (basic) Spacewalk client support to Debian. As soon as the packages passed NEW and #677871 was applied (thanks to the APT guys for working on that already), managing Debian clients with Spacewalk should work out of the box. Of course we also had a little keysigning party :)

Statistics
  • about 68 bugs in unstable/testing were triaged/patched/fixed or at least pinged
  • 54 bugs were tagged to show if they affect Squeeze, several other bugs were pinged to retrieve necessary information or to trigger an update in the next stable pointrelease.
  • 5 packages were introduced into Debian (still in NEW, though) - the Spacewalk client related packages and libapache2-mod-auth-memcookie.

Accomodation Thanks to Debian funds we were able to provide accomodation for four participants in the JUFA youth hostel in Salzburg. We had paid in advance for eight, but changing to rooms with a higher category for only 4 people would have been equally or more expensive.

Press/Media coverage Additionally to being mentioned in the calendars on ProLinux and similar pages, we had some press coverage by the local newspaper and online magazines:

Fun facts We consumed 2kg of Leberkas, a big plate of "Buchteln mit Vanillesosse", about 16000cm^2 of Pizza, about 80 litres of coke, juice, beer and wine and I guess we drank at least the same amount of water. We had coffee made of 1.5kg coffee beans and managed to empty the (formerly well filled) icemaker in the fridge. Also we had successful training sessions of a standard Debconf game (rules won't be explained here obviously). Maybe we even successfully spread the game to the employees of a commercial linux distribution ;)

22 June 2012

Tanguy Ortolo: We need you for the NTP pool!

Ferdinand Berthoud's marine clock #2 You already know NTP, the protocol that allows your computers to keep an accurate clock. Thanks to a global pool of time servers, it is available to everyone on the Internet. This public pool is constituted by volunteers' servers and it needs more.So, if you you should consider joining the pool. It is actually quite easy:
  1. install NTP;
  2. configure it to use five good servers from your region;
  3. register your server.

Alexander Reichle-Schmehl: Release Critical Bug report for Week 25

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:
In Total:1167
Affecting Wheezy:771
Wheezy only:189
Remaining to be fixed in Wheezy:582
Of these 582 bugs, the following tags are set:
Pending in Wheezy:31
Patched in Wheezy:82
Duplicates in Wheezy:57
Can be fixed in a security Update:21
Contrib or non-free in Wheezy:11
Claimed in Wheezy:2
Delayed in Wheezy:5
Otherwise fixed in Wheezy:47
Ignoring all the above (multiple tags possible) 376 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released. However, with the view of the Release Managers, 671 need to be dealt with for the release to happen. Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

Paul Tagliamonte: my quest to get desktop-base set before wheezy

I ve begun work on desktop-base to get it ready for wheezy. As many of you know, we saw the SpaceFun theme with Squeeze, and currently Wheezy as well. Rather then see the same theme twice, I ve taken the liberty of updating desktop-base to use the Joy theme proposed for Wheezy. My announcement can be seen on lists.debian.org, and details can be found there. Rather then drone on, check out what s been done so far. We ll be going from this: To: Game on!

21 June 2012

John Goerzen: I introduced my 5-year-old and 2-year-old to startx and xmonad. They re DELIGHTED!

Two years ago, Jacob (then 3) and I built his first computer together. I installed Debian on it, but never put a GUI on the thing. It s command-line, and has provided lots of enjoyment off and on over the last couple of years. I ve written extensively about what our boys like to do, and the delight they have at learning things on the command line. The looks of shock I get from people when I explain, as if it s perfectly natural, that my child has been able to log in by himself to a Linux shell since age 3, are amusing and astounding. Especially considering that it is really not that hard. Instead of learning how to run an Xbox, he s learned how to run bash. I like that. Lately, Jacob (now 5) hasn t been spending much time with it. He isn t really at a stage where he wants to push his limits too far, I think, but yet also gets bored with the familiar. So I thought it was time to introduce a GUI in a limited fashion, perhaps to let him download photos and video from his Vtech toy camera (that takes real low-res photos and videos which can be downloaded over a USB1 link). He s familiar with the concept, at least somewhat, having seen GUIs on Terah s computer (Gnome 2) and mine (xfce4 + xmonad). So last night, Oliver (age 2) and I went down to the basement on a mouse-finding expedition. Sure enough, I had an old PS/2 mouse down there that would work fine. The boys both helped string it through the desk up on our play room, and were tremendously excited to see the red light underneath it when the computer came on. Barely able to contain the excitement, really. A bit like I remember being when I got my first mouse (at a bit of an older age, I suppose.) I helped him them in as root for the very first time. (Jacob typed root , and I typed the password, and provided the explanation for why we were telling the computer we were root .) Jacob and Oliver alternated typing bits of some apt-get command lines. Then while we waited for software to download, I had to answer repeated questions of how soon will the mouse work? and what does install mean? Finally it was there, and I told Jacob to type startx. I intentionally did not install a display manager; more on that later. He pressed Enter, the screen went blank for about 5 seconds, and then X appeared. Excited can t begin to describe how they acted. They took turns playing with the mouse. They loved how the trash can icon (I started with XFCE) showed trash IN the trash can. But they are just learning the mouse, and there s a lot about a typical GUI that is unfriendly to someone that isn t yet proficient with a mouse. The close buttons are disappointingly small, things can be too easily dragged on and off the panel and menus. When I sat down to think about it, the typical GUI design does not present a very good it always works the same interface that would be good for a child. And then it occurred to me: the perfect GUI for a child would be simply xmonad (a tiling window manager that can be controlled almost entirely by keyboard and has no need for mouse movements in most cases.) No desktop environment, no file manager in the root window. Just a window manager in the classic X way. Of course! So after the boys were in bed, I installed xmonad. I gave Jacob s account a simple .xsession that starts a terminal and xmonad. Today, Jacob informed me that he wanted his computer to look just like yours. Playing right into my hands, that was! But when he excitedly typed startx, he said it wasn t just like mine. Uh oh. Turns out he wanted the same wallpaper as my computer uses. Whew. We found it, I figured out that xli(1) loads it in the root window, and so I added a third line to .xsession. More delight unlocked! Jacob mastered the basics of xmonad really quickly. Alt-Shift-C to close a window. Alt-Shift-Q to quit back to the big black screen . Alt-Shift-Enter to get a terminal window. We launched thunar (the XFCE file manager) and plugged in his camera. He had a good deal of fun looking at photos and videos from it. But then I dropped the true highlight of the day for him: I offered to install Tuxpaint for him. That s probably his favorite program of all time. He watched impatiently as apt-get counted down 1m30s for tuxpaint and its libraries. Then we launched it, and he wanted to skip supper so he could keep playing Tuxpaint on my VERY OWN COMPUTER! I d been debating how to introduce GUIs for a very long time. It has not escaped my attention that children that used Commodores or TRS-80s or DOS knew a lot more about how their computers worked, on average, than those of the same age that use Windows or MacOS. I didn t want our boys to skip an entire phase of learning how their technology works. I am pleased with this solution; they still run commands to launch things, yet get to play with more than text-based programs. At bedtime, Jacob asked me, very seriously: Dad, how do I start tuxpaint again? First you log in and type startx. Then you can use the mouse. Jacob nods, a contemplative look on his face.. Then, I continue, you type tuxpaint in the terminal, and it comes right up. Jacob nodded very seriously a second time, as if committing this very important information to long-term memory. Then gave a single excited clap, yelled Great! , and dashed off.

20 June 2012

DebConf team: Call for DebConf14 venues! (Posted by Gunnar Wolf)

2000, Bordeaux, France. 2001, Bordeaux, France. 2002, Ottawa, Canada. 2003, Oslo, Norway. 2004, Porto Alegre, Brazil. 2005, Helsinki, Finland. 2006, Oaxtepec, Mexico. 2007, Edinburgh, United Kingdom. 2008, Mar del Plata, Argentina. 2009, C ceres, Spain. 2010, New York, United States. 2011, Banja Luka, Bosnia and Herzegovina. 2012, Managua, Nicaragua. 2013, Vaumarcus, Switzerland. We have gone to all different latitudes (although not yet longitudes!). We have had very hot and very cold weather. Great cities and small villages have hosted us. What will be the next addition to this great list? As we approach DebConf12 (10 days to DebCamp and counting, yay!), we should keep the future in mind. So, as every year in recent history, we will once again have a DebConf session presenting possible venues for the next year. Organizing DebConf in your city means a lot of hard work. It also means one of the greatest personal experiences you can imagine. And it is a great way to contribute to Debian. The decision process for every DebConf venue starts two years before, with presentations during DebConf(n-2). That means, if you consider presenting a bid for DebConf14, now is the moment to act! Do you have to be present at Managua to propose your bid? No. You can proxy via somebody I d suggest to do it via somebody who knows the location you are suggesting, but basically, choose a friend that you trust that trusts you. Of course, you can participate in the presentation session via IRC. Do you have to be a Debian Developer to propose a bid? No. For DebConf9, none of the C ceres guys was a DD; for DebConf10, some of the people most involved from the local New Yorkers were not DDs. For DC11, none of our hosts in Bosnia are DDs. And for DC12, the dear and overworked Nicaraguan crew is also made from people interested in getting closer to the Debian project, but not DDs. Do you have to decide now? No. This is just a call for a first presentation, but the decision regarding DC14 will be taken probably around March 2013. However, giving a nice presentation at DebConf helps a lot, gives you visibility, and will get the ball rolling. Is there a geographical bias? Slight. So far, and since the second DebConf, we have kept the tradition not to repeat continents on two successive DebConfs. This is not a hard condition, however! While there is some probability that DebConf14 will be somewhere in America (the continent, of course), it depends on the proposals more than on any pseudo-rule. What do you need to start thinking about? Go visit our prospective location checklist. You can also look at what other teams have historically presented. And of course, go to the DebConf14 wiki planning page Register there, even if you are just in the early phases of finding data.

Andrew Pollock: [life] On running, Android apps, personal area networks and fund raising

Sarah and I are running in the 5K Packard Summer Scamper on Saturday. Sarah's been working on her fitness for a few months now, and I belatedly started working on mine about a 6 weeks ago. (Sarah's fund raising, so if you'd like to sponsor her, her page is here) I've been using the Couch to 5K program to get my fitness up, but I haven't managed to stick to it exactly, due to a combination of injury and wet weather. I have still managed to get myself to the point where I can run for 28 minutes solidly, which is close to 5K (I'm doing my last training run tomorrow, so I'll see what the distance comes out to). I've been doing up to three runs a week at lunchtime around the Shoreline Amphitheatre area, using my Galaxy Nexus and some Bluetooth headphones, and I've got a pretty nice little setup. I'm using Pandora for some music to listen to, C25K Pro for the actual running intervals. It has voice prompts that tell me when to run, or to start a walk interval. Finally, I'm using Strava Run for recording the run and getting some additional statistics during the run (kilometre markers, split times and time elapsed). These also come through as voice prompts. I'm not currently wearing a watch, and I want to keep track of the wall time, so I've also found Audio Clock, which verbally announces the time (I've set it to do it every 15 minutes that a Bluetooth headset is connected). What I really like about all of this is that I get spoken voice prompts and information, seamlessly mixed in with the music from Pandora, all while I'm running. I've got my phone strapped to my arm, and I don't have to look at it during the run, and there's no wires. I'm a huge personal metrics junkie, and that was why I previously eschewed running outdoors. I wanted that information on the treadmill display so I could push myself that little bit further. Sure, I'm not getting the constant information as much as if I had a treadmill display in front of me the whole time, but I'll settle for running in the great outdoors and getting some feedback every kilometre (if I get really desperate, Strava will do it every half kilometre). Not to diss optical augmented reality efforts too much, but I'd rather go for a combination of discreet audio announcements through a Bluetooth earpiece, perhaps complemented with something I choose to look at like a Pebble watch, than an in-your-face pair of glasses. My Bluetooth headphones are the around the back of the head variety that aren't going to bounce off during a run. I've really enjoyed doing the runs during the week, and when I was doing the full 3 runs a week of a given week of the Couch to 5K program, I'd really notice a difference by run number 3. I'm currently planning on sticking with doing a 5K run hopefully at least twice a week after this weekend. It's certainly done wonders for my waistline.

Russell Coker: SASL Authentication and Debian/Wheezy

After upgrading a mail server to Debian/Unstable (which will soon be released as Wheezy) I started getting SASL errors. 535 5.7.8 Error: authentication failed: no mechanism available The SMTP protocol gave the above error for both LOGIN and PLAIN methods. SASL LOGIN authentication failed: no mechanism available The postfix/smtpd process logged messages like the above in syslog. It turned out that the auxprop_plugin: mysql line had to be removed and replaced with the following two lines due to a change in the way SQL plugins are managed: auxprop_plugin: sql
sql_engine: mysql Also the SQL query needed to have %u replaced with %u@%r because we now have user and realm provided separately. Related posts:
  1. MySQL security in Debian Currently there is a problem with the MySQL default install...
  2. Kernel issues with Debian Xen and CentOS Kernels Last time I tried using a Debian 64bit Xen kernel...
  3. new release of postal Today I have released a significant new version of my...

19 June 2012

Neil Williams: subversion 1.7.5 insane upgrade requirement wasting days and days of effort

Upgrading the local version of subversion against a large subversion repository has so far taken three DAYS. It goes through this multi-gigabyte repository in no perceivable order, it goes through every single directory in every single tag and every single branch and refuses to run in any sub-directory. (Even after clearing out some older tags, the tags/ directory is still nearly 8Gb and that doesn't include any binary files.) Each tag within tags/ is 500Mb and contains nearly 6000 sub-directories. Yes, it's large but svn upgrade should still be able to handle it without crippling every machine.

I can't image how svn is going to manage when the server finally gets upgraded. Probably be quicker to dump the entire repo and reimport it.

When I finally give up the will to proceed or simply need to use this LAPTOP for something else, I have to interrupt it with Ctrl-C at which point is starts all over again!

This isn't a slow machine, it's a i5 quad-core T410 with 8Gb of RAM and 1Tb of storage - and svn has made it crawl for days. The only way to pair down the working copy is to delete every tag and branch which means losing data like old build logs and old packages. The repository is this large because it's tracking the development of multiple commercial products which share common code but which also have numerous releases and release updates.

I can't even use svn st on this repo without this completing - I haven't been able to work on this repo since this started. Absolutely insane.

Pondering filing an RC bug on the basis of unjustifiable data loss. We have many machines at work with this repository checked out and if we ever migrate to Wheezy, it's going to mean a WEEK of lost work!!

subversion                    1.7.5-1


So, a warning for anyone else using subversion 1.7.5 with a very large repository: DELETE ALL TAGS and BRANCHES and any other directory anywhere in every single working copy tree on every machine which ever wants to use that copy again before even thinking about upgrading to 1.7.5.

The tags directories don't even need to be upgraded because we only use those to rebuild the code as it was in chroots.

Unspeakably furious about such a completely dumb tool being thrown into the mix DAYS before the Wheezy freeze. Now I have to kill it AGAIN just so that I can suspend the laptop. IDIOTS.

DebConf team: Reconfirmation deadline tomorrow (Posted by Holger Levsen)

Just a quick reminder: if you haven t reconfirmed in our beloved pentabarf system, that you ll be attending DebConf12 in Managua in just a bit more then 2 weeks, then do so now! The reconfirmation deadline is tomorrow!

Joey Hess: I work for The Internet now

I have an interesting problem: How do I shoehorn "hired by The Internet for a full year to work on Free Software" into my resume? Yes, the git-annex Kickstarter went well. :) I had asked for enough to get by for three months. Multiple people thought I should instead work on it for a full year and really do the idea justice. Jason Scott was especially enthusiastic about this idea. So I added new goals and eventually it got there. graph: up and to the right, to 800% funded Don Marti thinks the success of my Kickstarter validates crowdfunding for Free Software. Hard to say; this is not the first Free Software to be funded on Kickstarter. Remember Diaspora? Here's what I think worked to make this a success: I've already put two weeks work into developing the git-annex assistant. I'm blogging about my progress every day on its development blog. The first new feature, a git annex watch command that automatically commits changes to git, will be in the next release of git-annex. I can already tell this is going to be a year of hard work, difficult problems, and great results. Thank you for helping make it happen.

Sorina Sandu: Birthday present

Hello!

Yesterday was my birthday, I am 21 years old now :D

Here's what my friends got me as a gift:


One alternative explanation that I have received for this, apart from working on unstable, is that... women are generally unstable :P

Jo Shields: Enormity

(This blog post is a more firm version of a series of tweets and forum posts I made a few weeks ago. This should also be considered a refresh of this post by Mirco Bauer a few years ago) It has been said that Mono is bloated, and that people should use lighter frameworks that don t pull in hundreds of meg . How much basis in reality is there for those comments? Well, let s do a little thought exercise. How much space is the minimum space needed for several languages to work? For this experiment, we will be looking at the required installation size on disk of several language frameworks, each time installing the bare minimum for a command-line hello world app in that language to run. For example, the Ruby result is the minimum install required to run:
puts 'Hello world'
and the Mono result is the minimum install required to run the following code, which has been compiled on a different machine using dmcs :
public class Hello1
 
   public static void Main()
    
      System.Console.WriteLine("Hello, World!");
    
 
Obviously, for dynamic languages like Ruby and Python, there is no compiler step needed. For languages like C# and Java, it is fair to compile these on a different machine as end-users of packages running these frameworks receive binaries, not source they do not need a development environment. So, on to the test environment. This is a bare bootstrapped AMD64 Debian Unstable system, as of today (i.e. debootstrap sid /tmp/badger ). Its install size is 275MB. So how much space? Ruby 1.9 Test code is as follows:
puts 'Hello world'
Install requirements are as follows:
root@dream:/# aptitude install ruby1.9.1
The following NEW packages will be installed:
  libffi5 a  libruby1.9.1 a  libyaml-0-2 a  ruby1.9.1 
0 packages upgraded, 4 newly installed, 0 to remove and 0 not upgraded.
Need to get 4703 kB of archives. After unpacking 13.1 MB will be used.
OpenJDK 7 Test code is as follows:
class HelloWorldApp  
    public static void main(String[] args)  
        System.out.println("Hello world");
     
 
Install requirements are as follows:
root@dream:/# aptitude install openjdk-7-jre-headless
The following NEW packages will be installed:
  ca-certificates a  ca-certificates-java a  dbus a  fontconfig-config a  
  icedtea-7-jre-cacao a  icedtea-7-jre-jamvm a  java-common a  
  krb5-locales a  libavahi-client3 a  libavahi-common-data a  
  libavahi-common3 a  libcap2 a  libcups2 a  libdbus-1-3 a  libexpat1 a  
  libffi5 a  libfontconfig1 a  libfreetype6 a  libglib2.0-0 a  
  libglib2.0-data a  libgssapi-krb5-2 a  libjpeg8 a  libk5crypto3 a  
  libkeyutils1 a  libkrb5-3 a  libkrb5support0 a  liblcms2-2 a  libnspr4 a  
  libnss3 libnss3-1d a  libpcre3 a  libpcsclite1 a  libsystemd-login0 a  
  libxml2 a  openjdk-7-jre-headless openjdk-7-jre-lib a  openssl a  
  sgml-base a  shared-mime-info a  ttf-dejavu-core a  tzdata-java a  ucf a  
  xml-core a  
0 packages upgraded, 43 newly installed, 0 to remove and 0 not upgraded.
Need to get 51.5 MB of archives. After unpacking 137 MB will be used.
Python 3.2 Test code is as follows:
print ("Hello world")
Install requirements are as follows:
root@dream:/# aptitude install python3.2-minimal
The following NEW packages will be installed:
  file a  libexpat1 a  libffi5 a  libmagic1 a  mime-support a  python3.2 a  python3.2-minimal 
0 packages upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 4853 kB of archives. After unpacking 17.7 MB will be used.
Mono 2.10 (C#) Test code is as follows:
public class Hello
 
   public static void Main()
    
      System.Console.WriteLine("Hello world");
    
 
Install requirements are as follows:
root@dream:/# aptitude install mono-runtime
The following NEW packages will be installed:
  binfmt-support a  cli-common a  libmono-corlib4.0-cil a  libmono-i18n-west4.0-cil a  libmono-i18n4.0-cil a  
  libmono-security4.0-cil a  libmono-system-configuration4.0-cil a  libmono-system-security4.0-cil a  
  libmono-system-xml4.0-cil a  libmono-system4.0-cil a  mono-4.0-gac a  mono-gac a  mono-runtime 
0 packages upgraded, 13 newly installed, 0 to remove and 0 not upgraded.
Need to get 4383 kB of archives. After unpacking 11.5 MB will be used.
Of course, every time I try to make the argument that Mono is much leaner than what people propose as an alternative, the anti-Mono brigade go on the defensive and insist that nobody in their right minds uses Python or Ruby either everything ever should be written in Qt. So, hypothetically Qt 4.8 (C++) Test code is as follows:
#include <QTextStream>
int main(int argc, char **argv)
 
   QTextStream s(stdout);
   s << "Hello, world!\n";
   return 0;
 
Install requirements are as follows:
root@dream:/# aptitude install libqtcore4
The following NEW packages will be installed:
  libffi5 a  libglib2.0-0 a  libglib2.0-data a  libpcre3 a  libqtcore4 
  libxml2 a  sgml-base a  shared-mime-info a  xml-core a  
0 packages upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.3 MB of archives. After unpacking 27.8 MB will be used.
Presented without comment for your consideration. Edit: There are complaints that these numbers are misleading, because they show default package manager behaviour, rather than what happens when passing extra flags or config file settings to Apt. When disabling installation of Recommends: packages, the numbers change as follows:

18 June 2012

Gunnar Wolf: Turning failure into apparent success, and carrying on: e-voting in Jalisco

I will sound monothematic, but I have been devoting quite a bit of work to this topic lately: Trying to stop the advance of e-voting in Mexico, Latin America and the world. Why trying to stop it? Isn't technology supposed to help us, to get trustable processes? Yes, it's supposed to... but it just cannot achieve it, no matter how hard it is tried I won't get into explanations in this blog post, but there is plenty of information. Feel free to ask me for further details. Anyway Yesterday (Sunday, 2012-06-17) was the fifth simulated voting that will lead to the first wide-scale deployment of electronic voting booths in my country: About 10% of the population of the state of Jalisco (that means, ~500,000 people) will cast their votes on July 1st electronically. This particular case illustrates how simulated votings can be used to forge a lie: Pounce Consulting, the company that won the e-voting project for IEPC (Jalisco's voting authority), delivered their booths over 40 days late, just before the deadline for the project to be canceled. Oh, and by the way, it's the same company that just failed to deliver on time for another planned local authority (10% of the booths in the Federal District, where I live, where fortunately 100% of the votes will be cast on traditional, auditable and cheap paper). After this delay, five voting simulations were programmed, to get the local population acquinted with them. The first ones just failed to get the population's interest and had close to 40% failure rates (mainly regarding transmission). Several other "minor details" were reported, including mechanical details that allowed subsequent voters to see the vote of who had just left. Anyway, making long story short: The fifth and last simulation was held yesterday. Officially, it was finally successful (about time). As these booths include the "facilities" to communicate the results via the cellular network, but the populations where they are to be deployed do not yet have cellular coverage, 10% of the booths will have to be carried back to the Districtal Header (that can be a ~10hr trip) to be counted. Also, in all places, traditional paper stationery and paraphernalia will be printed just in case it is needed (and when will they now? When half of the votes are cast and lost?) Anyway... e-voting is still in its first stage in Mexico. Right now, I'm sure, no attempts to rig the election will be made (centrally). But every effort will be made (as it has been made) to dismiss the obviously big and nontrivial ways it has failed and will fail, and any problems will be labeled as "minor". And probably by 2018 we will be facing many more states (even nationwide) deployments. But propaganda fails to see the obvious: E-voting is more expensive, more complicated, leads to more possible failure states. E-voting should not be deployed in large-scale (i.e. more than a couple of hundred voters) elections. Electronic voting is insecure, violates secrecy, allows for fraud. No matter how many locks are put into it.

17 June 2012

Johannes Schauer: port bootstrap build-ordering tool report 2

A copy of this post is sent to soc-coordination@lists.alioth.debian.org as well as to debian-bootstrap@lists.mister-muffin.de. Diary June 4 I added the first version of basenocycles.ml to git. Given an initial set of cross built packages, it tries to compile as much as possible on the resulting system in multiple rounds. June 5 During June 3, I discovered and error in my program that would only come up when using the Debian Sid package lists as the input:
Fatal error: exception Assert_failure("common/edosSolver.ml", 610, 11)
On this day, June 5, I wrote a minimal test case for this problem. The same day, Pietro figured out that this is a bug in dose which will be fixed in the next release. Begin writing code to figure out how important a binary package is for the further build process. Try to use Depsolver.edos_install to find out what packages are needed to make debhelper available. Restructure basenocycles.ml, exclude source packages that already have been built, still trouble with already existing binary packages and Cudf.mem_installed, comment stuff better. June 6 I wrote some crude code (only estimate, not correct, fixed later) that would give a rough overview of how often a given binary package is directly required as a build dependency. Debhelper came out as the most needed package. It is architecture:all, so it does not have to be built but it has unfulfilled runtime dependencies. To make those packages available, 13 (actually 11, fixed later) packages have to be compiled on Ubuntu Natty. But those packages all (except gettext) require debhelper itself to be built. The first dependency cycle. This dependency cycle (actually, the 12 cycles) can be broken by either cross compiling those source packages or by making them build without debhelper. One goal of the program is to help decide what the easier option is, but this is not yet implemented. To play around a bit, I created the possibility to specify a list of packages that are additionally to the minimal set of cross compiled packages also cross compiled. I added the 13 packages found above to the list, thus making the binary packages they build available. This made debhelper available in the system. As a result, 1625 out of 3339 source packages can be built with just a minimal build system (priority:essential packages plus build-essential) and debhelper available. The next package that blocks the most other source packages from being built is cdbs. The next nine packages in that list also require cdbs so it seems to be the next important package to be available. Pietro's suggestions make me:
 - do not open BootstrapCommon but ExtLib, Common, Algo, Debian
 - do proper option parsing and logging
 - use Debcudf.ignore_essential = true
 - do Debcudf.init_tables (binlist@srclist)
 - use @ with shorter list first
 - use more List.rev_append instead of @
 - use CudfAdd.who_provides to find if a package is available
June 7 Pietro and Patrick suggest that for solving the debhelper cycles, one could provide a minimal debhelper version so that the above list of 12 packages can be built without debhelper. I try to figure out how to get a list of packages that are missing to make a package installable/buildable. This functionality should be provided in dose but I fail to find it. June 8 Lacking a solution of the problem of June 7, I write a mail to Pietro. I start my first graphs in ocaml using the ocamlgraph library. The graph I generate, starts off at a binary package. For each binary package it connects new vertices as its runtime dependencies. If a binary package is not arch:all and also not yet otherwise compiled, its source package is also added. The result is a graph in which set of source packages in it will make the initial package available, if those source packages would be cross compiled. The graph is extended further than the source packages. June 9 I refine a couple of functions, make univ_get_pkg_by_name return the package with the highest version number. I wrote a rather lengthy (1027 words) email to the list that explains my status as of this day. I can create graphviz dot files with ocaml, can create node and edge types and create the graph by an imperative pattern that I saw a lot in Pietro's code. Disjunctions are not yet handled correctly (see mail from June 8). The graphs generated look like the following: http://mister-muffin.de/p/8nyc.png June 11 I write a test case which shows how CudfAdd.who_provides doesnt find virtual packages. Automate the process of finding the packages that, if cross compiled, would make another package available. Add more predicates (identifying source packages) and improve input file reading code. Move build_compile_rounds which compiles as many source packages as it can in multiple rounds on a given minimal system a toplevel function and thereby abstract it more. Create a rudimentary text based menu to choose different actions to take for an analysis. Start writing an extended version of simple_dependency_graph for deeper analysis. Use xdot to show graphs from the text menu. Allow saving those graphs to a file. June 12 Move functionality from the extended version of simple_dependency_graph over to the normal version and delete the extended version. Add the new Closure vertex type. Create extended_dependency_graph which is supposed to not contain single binary package vertices but handle a package and its installation set as one vertex. The output of extended_dependency_graph is optionally reduced to the biggest (non degenerate) strongly connected component. User gets the option of choosing the exploration depth. June 13 Pietro replies to my mail from June 8 but apparently I failed to express myself well enough in my last mail, so I rephrase my question. Pietro replies to my email from June 11 and explains how the effect I see is due to "a nuisance of the debian to cudf encoding". As a result I change my code accordingly. June 14 Another lengthy (1130 words) email to the list. I explain what was done in the past days, what parts work and how they work. I list some rationales on why I did things the way I did them. The most important observation is, that after improving my code again and again, I ended up representing the dependency cycle problem in the same (very similar) way that Pietro suggested in the beginning. This is probably a good discovery. Lots of data of that email is now of only little use as of June 16, I make lots of improvements in correctness. As I dont have an answer to my other email to Pietro from June 13, I implement a very crude way to get an answer to the question of what packages are missing for a package to be available/compileable. I called it flatten_vpkgformula_best_effort and it suffers from many faults including disjunctions and package conflicts. Patrick spots a problem. As a result, I make sure that at no point, the source package of an arch:all package can be listed. June 15 As a reply to my mail from June 13, Pietro creates a new branch in the git and adds the code I needed to get a proper installation set. June 16 As a result of Pietro's efforts from June 15, I make great advancements on all fronts. Details of the current status follow in the next section. Results A big leap was made on June 16 due to Pietro's great help on making me understand how Depsolver.listcheck can be used for my purposes. My difficulties in finding the solution myself are rooted in many parts of the dose framework being poorly commented but Pietro did already a couple of documentation commits whenever things were unclear for me. Using Depsolver.listcheck makes it possible to be more distribution agnostic and I dont have to handle vpkgs, virtual packages and constraints myself anymore. The code also doesnt suffer anymore by wrongly analyzed dependencies and conflicts. The only thing that is not yet taken care of, is that Depsolver.listcheck only chooses one out of several possible installation set. A final version should be able to take into account that a different installation set could provide a better solution. Overall, in comparison to two weeks ago, I can now properly build, traverse and analyze graphs, can choose an installation set properly, understand more about dependencies, closures, dose and ocaml in general. Finding the importance of binary packages for building When calculating how many source packages are depending on the availability of a binary package I originally flattened the pkg.Cudf.depends list twice for a rough overview. This is of course wrong due to disjunctions and conflicts and also doesnt provide a deep dependency inspection. The new method is to calculate an installation set that is necessary to compile a source package for every source package. The resulting list of binary packages is then used to find out how often a binary package appears in an installation set. I see three drawbacks though: Removing simple graph The simple graph which contained single binary and source packages was removed. I realized it doesnt really serve any purpose to look at it. As a result, Bin vertices and InstallDep edges are also not part of the graph anymore. Since it was responsible for generating the list of source packages that have to be cross built to make a package available, I created a new function get_cross_source_packages which uses an installation to serve the same purpose. Fix extended_dependency_graph extended_dependency_graph now uses installation sets for generating the list of packages that is needed to compile a source package or install a binary package. The list of build dependencies does not include packages that are already installable. The list of runtime dependencies does not include packages that are otherwise available (cross built, arch:all...). Instead of checking for list membership all the time, I created hash tables for the list of installable as well as for the list of available binary packages. Future There are two big tasks for the next two weeks: Task one is to find a way to give hints on which packages to best consider for having reduced build dependencies. This would then probably finally make use of Pietro's cycle algorithms. Task two is to find a way to break cycles and create a build-DAG from a list of packages that already have staged build dependency information. Patrick is currently working on patching dpkg with Build-Depends-StageN dependencies as making perl cross compilable. If he doesnt need the ability to decide which packages to modify to have staged build dependencies in the near future, then task one is probably less urgent and therefor of less importance right now? On the other hand, I can easily generate fake reduced build dependencies so that doing task two right now would not be a problem. Also, having the solution for task two will make it possible to show the user what effect it would have to add reduced build dependencies to a package. For the reasons above (it's not urgent, task one profits from task two being solved) I will go and implement task two first (if there are no objections from my mentors). Another idea, that I discussed with wookey and Patrick yesterday, was that due to multiarch being used for more and more packages, there should exist a set of packages that is cross compilable without any change to the package. We agreed that I make a list of packages that, if cross compiled, would break dependency cycles and make other packages available. I created such a list of about 160 packages for Ubuntu Natty that, if cross compiled, made it possible to have 87% of Natty available (those numbers have to be treated with caution as I did not yet use the proper way of installation sets when generating that list, but the order of magnitude should be correct). Wookey can then try to cross compile those packages. If some packages of those "crucial" source packages are found to be cross compilable, then they should be cross compiled because it means that no work has to be done to break some cycles. Cross compiling all packages that are cross compilable out of the box is no solution, as only natively compiled packages can go into the archive. This is why the list of potentially additionally cross compiled source packages has to be kept as small as possible.

16 June 2012

Jon Dowland: Debian Day #3

Once again Debian day was somewhat compromised: this time by more DIY. I still managed a few bits and pieces, amortized over the rest of the week. I benefited from some train time this weekend. A new version of flactag and catching up the flactag packaging vcs repository. A maintainer endorsement for one of the flactag maintainers and a request to NM for another maintainer to be granted access to the collab-maint repository. More work on the musicbrainz 'transition': First, usertag tracking of packages to update, with upstream bugs where necessary. Tested and confirmed Michael Biebl's sushi patch; found and fixed a sushi bug along the way; wrote patches for sound-juicer ( merged upstream) and goobox.

Next.

Previous.