Search Results: "ag"

19 April 2014

Russell Coker: Swap Space and SSD

In 2007 I wrote a blog post about swap space [1]. The main point of that article was to debunk the claim that Linux needs a swap space twice as large as main memory (in summary such advice is based on BSD Unix systems and has never applied to Linux and that most storage devices aren t fast enough for large swap). That post was picked up by Barrapunto (Spanish Slashdot) and became one of the most popular posts I ve written [2]. In the past 7 years things have changed. Back then 2G of RAM was still a reasonable amount and 4G was a lot for a desktop system or laptop. Now there are even phones with 3G of RAM, 4G is about the minimum for any new desktop or laptop, and desktop/laptop systems with 16G aren t that uncommon. Another significant development is the use of SSDs which dramatically improve speed for some operations (mainly seeks). As SATA SSDs for desktop use start at about $110 I think it s safe to assume that everyone who wants a fast desktop system has one. As a major limiting factor in swap use is the seek performance of the storage the use of SSDs should allow greater swap use. My main desktop system has 4G of RAM (it s an older Intel 64bit system and doesn t support more) and has 4G of swap space on an Intel SSD. My work flow involves having dozens of Chromium tabs open at the same time, usually performance starts to drop when I get to about 3.5G of swap in use. While SSD generally has excellent random IO performance the contiguous IO performance often isn t much better than hard drives. My Intel SSDSC2CT12 300i 128G can do over 5000 random seeks per second but for sustained contiguous filesystem IO can only do 225M/s for writes and 274M/s for reads. The contiguous IO performance is less than twice as good as a cheap 3TB SATA disk. It also seems that the performance of SSDs aren t as consistent as that of hard drives, when a hard drive delivers a certain level of performance then it can generally do so 24*7 but a SSD will sometimes reduce performance to move blocks around (the erase block size is usually a lot larger than the filesystem block size). It s obvious that SSDs allow significantly better swap performance and therefore make it viable to run a system with more swap in use but that doesn t allow unlimited swap. Even when using programs like Chromium (which seems to allocate huge amounts of RAM that aren t used much) it doesn t seem viable to have swap be much bigger than 4G on a system with 4G of RAM. Now I could buy another SSD and use two swap spaces for double the overall throughput (which would still be cheaper than buying a PC that supports 8G of RAM), but that still wouldn t solve all problems. One issue I have been having on occasion is BTRFS failing to allocate kernel memory when managing snapshots. I m not sure if this would be solved by adding more RAM as it could be an issue of RAM fragmentation I won t file a bug report about this until some of the other BTRFS bugs are fixed. Another problem I have had is when running Minecraft the driver for my ATI video card fails to allocate contiguous kernel memory, this is one that almost certainly wouldn t be solved by just adding more swap but might be solved if I tweaked the kernel to be more aggressive about swapping out data. In 2007 when using hard drives for swap I found that the maximum space that could be used with reasonable performance for typical desktop operations was something less than 2G. Now with a SSD the limit for usable swap seems to be something like 4G on a system with 4G of RAM. On a system with only 2G of RAM that might allow the system to be usable with swap being twice as large as RAM, but with the amounts of RAM in modern PCs it seems that even SSD doesn t allow using a swap space larger than RAM for typical use unless it s being used for hibernation. Conclusion It seems that nothing has significantly changed in the last 7 years. We have more RAM, faster storage, and applications that are more memory hungry. The end result is that swap still isn t very usable for anything other than hibernation if it s larger than RAM. It would be nice if application developers could stop increasing the use of RAM. Currently it seems that the RAM requirements for Linux desktop use are about 3 years behind the RAM requirements for Windows. This is convenient as a PC is fully depreciated according to the tax office after 3 years. This makes it easy to get 3 year old PCs cheaply (or sometimes for free as rubbish) which work really well for Linux. But it would be nice if we could be 4 or 5 years behind Windows in terms of hardware requirements to reduce the hardware requirements for Linux users even further.

Russell Coker: Phone Based Lectures

Early this month at a LUV meeting I gave a talk with only my mobile phone to store notes. I used Google Keep to write the notes as it s one of the easiest ways of writing a note on a PC and quickly transferring it to a phone if I keep doing this I will find some suitable free software for this task. Owncloud seems promising [1], but at the moment I m more concerned with people issues than software. Over the years I ve experimented with different ways of presenting lectures. I m now working with the theory that presenting the same data twice (by speaking and text on a projector) distracts the audience and decreases learning. Editing and Viewing Notes Google Keep is adequate for maintaining notes, it s based on notes that are a list of items (like a shopping list) which is fine for lecture notes. It probably has lots of other functionality but I don t care much about that. Keep is really fast at updating notes, I can commit a change on my laptop and have it visible on my phone in a few seconds over 3G. Most of the lectures that I ve given have involved notes on a laptop. My first laptop was a Thinkpad 385XD with a 12.1 display and all my subsequent laptops have had a bigger screen. When a laptop with a 12 or larger screen is on a lectern I can see the notes at a glance without having to lean forward when 15 or fewer lines of text are displayed on the screen. 15 lines of text is about the maximum that can be displayed on a slide for the audience to read and with the width of a computer display or projector is enough for a reasonable quantity of text. When I run Keep on my Galaxy Note 2 it displays about 20 rather short lines of text in a portrait orientation (5 points for a lecture) and 11 slightly longer lines in a landscape orientation (4 points). In both cases the amount of text displayed on a screen is less than that with a laptop while the font is a lot smaller. My aim is to use free software for everything, so when I replace Keep with Owncloud (or something similar) I will probably have some options for changing the font size. But that means having less than 5 points displayed on screen at a time and thus a change in the way I present my talks (I generally change the order of points based on how well the audience seem to get the concepts so seeing multiple points on screen at the same time is a benefit). The Samsung Galaxy Note 2 has a 5.5 display which is one of the largest displays available in a phone. The Sony Xperia X Ultra is one of the few larger phones with a 6.44 display that s a large phone but still not nearly large enough to have more than a few points on screen with a font readable by someone with average vision while it rests on a lectern. The most obvious solution to the problem of text size is to use a tablet. Modern 10 tablets have resolutions ranging from 1920*1080 to 2560*1600 and should be more readable than the Thinkpad I used in 1998 which had a 12 800*600 display. Another possibility that I m considering is using an old phone, a Samsung Galaxy S weighs 118 to 155 grams and is easier to hold up than a Galaxy Note 2 which weighs 180g. While 60g doesn t seem like much difference if I m going to hold a phone in front of me for most of an hour the smaller and lighter phone will be easier and maybe less distracting for the audience. Distributing URLs When I give a talk I often want to share the addresses of relevant web sites with the audience. When I give a talk with the traditional style lecture notes I just put the URLs on the final page (sometimes using tinyurl.com) for people to copy during question time. When I use a phone I have to find another way. I did a test with QR code recognition and found that a code that takes up most of the width of the screen of my Galaxy Note 2 can be recognised by a Galaxy S at a distance of 50cm. If I ran the same software on a 10 tablet then it would probably be readable at a distance of a meter, if I had the QR code take up the entire screen on a tablet it might be readable at 1.5m away, so it doesn t seem plausible to hold up a tablet and allow even the first few rows of the audience to decode a QR code. Even if newer phones have better photographic capabilities than the Galaxy S that I had available for testing there are still lots of people using old phones who I want to support. I think that if QR codes are to be used they have to be usable by at least the first three rows of the audience for a small audience of maybe 50 people as that would allow everyone who s interested to quickly get in range and scan the code at the end. Chris Samuel has a photo (taken at the same meeting) showing how a QR code from a phone could be distributed to a room [2]. But that won t work for all rooms. One option is to just have the QR code on my phone and allow audience members to scan it after the lecture. As most members of the audience won t want the URLs it should be possible for the interested people to queue up to scan the QR code(s). Another possibility I m considering is to use a temporary post on my documents blog (which isn t syndicated) for URLs. The WordPress client for Android works reasonably well so I could edit the URL list at any time. That would work reasonably well for talks that have lots of URLs which is quite rare for me. A final option is to use Twitter, at the end of a talk I could just tweet the URLs with suitable descriptions. A good portion of the Tweets that I have written is URLs for web sites that I find interesting so this isn t a change. This is probably the easiest option, but with the usual caveat of using a proprietary service as an interim measure until I get a free software alternative working. Any suggestions? Please comment if you have any ideas about ways of addressing these issues. Also please let me know if anyone is working on a distributed Twitter replacement. Please note that anything which doesn t support followers on multiple servers and re-tweets and tweeting to users on other servers isn t useful in this regard.

Thomas Goirand: OpenStack 2014.1, aka Icehouse, is out

The new version of OpenStack is out, and I have just finished uploading it all into Debian Sid. With a total of 38 packages that I uploaded yesterday (which was exhausting!), most, if not all, were only moving from Experimental to Sid with only tiny updates, and this represents the achievement of 6 months of packaging work. The new feature list is impressive, and I would like to highlight some part of it: For the moment, I haven t packaged Sahara (eg: Hadoop as a service), but it might come later as a customer of us might require it. There s a lot less unit tests issues in the packages I uploaded to Sid: all SQLAlchemy issues have been dealt with. I wasn t confident with the Havana release that Sid / Testing would be a good environment for OpenStack, but this time with Icehouse, I think it should be much better. Please test this brand new release and report issues on the BTS. As always, the packages are available also as Wheezy backports through the usual channels (see the official install guide).

Ben Hutchings: Linux kernel update for Debian 7.5; new Intel Ethernet drivers

Debian 7.5 will include an update to the Linux kernel, based on Linux 3.2.57. Package version 3.2.57-2 is currently available in the wheezy-proposed-updates suite. I would appreciate any testing people can do to find regressions in the next few days. In addition to bug fixes, this version updates the e1000e and igb drivers. The drivers are now based on the versions found in Linux 3.13, which support several newer chips (i210, i211, i217, i218, i354). Please consider testing this new kernel if you have an Intel gigabit Ethernet controller, even if it was already supported in Linux 3.2.

Paul Tagliamonte: Hy at PyCon 2014

I gave a talk this year at PyCon 2014, about one of my favorite subjects: Hy. Many of my regular readers will have no doubt explored Hy's thriving GitHub org, played with try-hy, or even installed it locally by pip installing it. I was lucky enough to be able to attend PyCon on behalf of Sunlight, with a solid contingint of my colleagues. We put together a writeup on the Sunlight blog if anyone was interested in our favorite talks.
Tons of really amazing questions, and such an amazingly warm reception from so many of my peers throughout this year's PyCon. Thank you so much to everyone that attended the talk. As always, you should Fork Hy on GitHub, follow @hylang on the twitters, and send in any bugs you find! Hopefully I'll be able to put my talk up in blog-post form soon, but until then feel free to look over the slides or just watch the talk. An extra shout-out to @akaptur for hacking on Hy during the sprints, and giving the exception system quite the workthrough. Thanks, Allison!

18 April 2014

Thorsten Glaser: Stay off my computer, puppet!

I was out, seeing something that wasn t there yet when I was at school (the web was not ubiquitous, back then), and decided to have a look:
pageok

Ugh. Oh well, PocketIE doesn t provide a View Source thingy, so I asked Natureshadow (who got the same result on his Android, and had no View Source either apparently, so he used cURL to see it). We saw (here, re-enacted using ftp(1)):

	tg@blau:~ $ ftp -Vo - http://www.draitschbrunnen.de/
	<!-- pageok -->
	<!-- managed by puppet -->
	<html>
	<pre>pageok</pre>
	</html>
 

This is the final straw after puppet managed to trash a sudoers(5) at work (I warned people to not introduce it) now it breaks websites.

(Of course, tools are useful, but at best to the skill of their users. Merely dumbly copying recipes from the net without any understanding just makes debugging harder for those of us with skills.) ObQuestion: Does anyone have a transcript (into UTF-8) and a translation for the other half of the OpenBSD 2.8 poster? (I get asked this regularily.)
Update: One person sent me the Kanji and Kana for it in UTF-8 , and they and one more person told me it s Hands off my machine! or Don t lay a hand on my machine! . Now I m not studying Japanese, but it LGTM in FixedMisc [MirOS], and JMdict from MirPorts says: ore no mashin ni te (w)o dasu na (roughly: my machine; particle; hands; particle; put out; prohibition) Thanks all, now I know what to tell visitors who wonder about that poster on my wall. ObTip: I can install a few hundred Debian VMs at work manually before the effort needed to automate d-i would amortise. So I decided not to. Coworkers are shocked. I keep flexibility (can decide to have machines differ), and the boss accepts my explanations. Think before doing automation just for the sake of automation!

Richard Hartmann: higher security

Instant classic
Trusted:
NO, there were errors:
The certificate does not apply to the given host
The certificate authority's certificate is invalid
The root certificate authority's certificate is not trusted for this purpose
The certificate cannot be verified for internal reasons
Signature Algorithm: md5WithRSAEncryption
    Issuer: C=XY, ST=Snake Desert, L=Snake Town, O=Snake Oil, Ltd, OU=Certificate Authority, CN=Snake Oil CA/emailAddress=ca@snakeoil.dom
    Validity
        Not Before: Oct 21 18:21:51 1999 GMT
        Not After : Oct 20 18:21:51 2001 GMT
    Subject: C=XY, ST=Snake Desert, L=Snake Town, O=Snake Oil, Ltd, OU=Webserver Team, CN=www.snakeoil.dom/emailAddress=www@snakeoil.dom
...
            X509v3 Subject Alternative Name: 
            email:www@snakeoil.dom

For your own pleasure:
openssl s_client -connect www.walton.com.tw:443 -showcerts

or just run
echo '
-----BEGIN CERTIFICATE-----
MIIDNjCCAp+gAwIBAgIBATANBgkqhkiG9w0BAQQFADCBqTELMAkGA1UEBhMCWFkx
FTATBgNVBAgTDFNuYWtlIERlc2VydDETMBEGA1UEBxMKU25ha2UgVG93bjEXMBUG
A1UEChMOU25ha2UgT2lsLCBMdGQxHjAcBgNVBAsTFUNlcnRpZmljYXRlIEF1dGhv
cml0eTEVMBMGA1UEAxMMU25ha2UgT2lsIENBMR4wHAYJKoZIhvcNAQkBFg9jYUBz
bmFrZW9pbC5kb20wHhcNOTkxMDIxMTgyMTUxWhcNMDExMDIwMTgyMTUxWjCBpzEL
MAkGA1UEBhMCWFkxFTATBgNVBAgTDFNuYWtlIERlc2VydDETMBEGA1UEBxMKU25h
a2UgVG93bjEXMBUGA1UEChMOU25ha2UgT2lsLCBMdGQxFzAVBgNVBAsTDldlYnNl
cnZlciBUZWFtMRkwFwYDVQQDExB3d3cuc25ha2VvaWwuZG9tMR8wHQYJKoZIhvcN
AQkBFhB3d3dAc25ha2VvaWwuZG9tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKB
gQC554Ro+VH0dJONqljPBW+C72MDNGNy9eXnzejXrczsHs3Pc92Vaat6CpIEEGue
yG29xagb1o7Gj2KRgpVYcmdx6tHd2JkFW5BcFVfWXL42PV4rf9ziYon8jWsbK2aE
+L6hCtcbxdbHOGZdSIWZJwc/1Vs70S/7ImW+Zds8YEFiAwIDAQABo24wbDAbBgNV
HREEFDASgRB3d3dAc25ha2VvaWwuZG9tMDoGCWCGSAGG+EIBDQQtFittb2Rfc3Ns
IGdlbmVyYXRlZCBjdXN0b20gc2VydmVyIGNlcnRpZmljYXRlMBEGCWCGSAGG+EIB
AQQEAwIGQDANBgkqhkiG9w0BAQQFAAOBgQB6MRsYGTXUR53/nTkRDQlBdgCcnhy3
hErfmPNl/Or5jWOmuufeIXqCvM6dK7kW/KBboui4pffIKUVafLUMdARVV6BpIGMI
5LmVFK3sgwuJ01v/90hCt4kTWoT8YHbBLtQh7PzWgJoBAY7MJmjSguYCRt91sU4K
s0dfWsdItkw4uQ==
-----END CERTIFICATE-----
'   openssl x509 -noout -text

At least they're secure against heartbleed.

17 April 2014

Elena 'valhalla' Grandi: DUCC-IT

There is exactly one month left before DUCC-IT, the Debian Ubuntu Community Conference Italia: a great chance to meet your free software developing neighborhoods.

This year it will be just one day, in Cesena, and it will include events targeted to both the community and a wider public.

The Call for Paper is still open, but only for a few days, so if you want to propose a talk/session hurry up!

#duccit14 @Debian

Andrew Pollock: [life] Day 79: Magic, flu shots, and play dates and dinner

Zoe slept until 7:45am this morning, which is absolutely unheard of in our house. She did wake up at about 5:15am yelling out for me because she'd kicked her doona off and lost Cowie, but went back to sleep once I sorted that out. She was super grumpy when she woke up, which I mostly attributed to being hungry, so I got breakfast into her as quickly as possible and she perked up afterwards. Today there was a free magic show at the Bulimba Library at 10:30am, so we biked down there. I really need to work on curbing Zoe's procrastination. We started trying to leave the house at 10am, and as it was, we only got there with 2 minutes to spare before the show started. Magic Glen put on a really good show. He was part comedian, part sleight of hand magician, and he did a very entertaining show. There were plenty of gags in it for the adults. Zoe started out sitting in my lap, but part way through just got up and moved closer to the front to sit with the other kids. I think she enjoyed herself. I'd have no hesitation hiring this guy for a future birthday party. Zoe had left her two stuffed toys from the car at Megan's house on Tuesday after our Port of Brisbane tour, and so after the magic show we biked to her place to retrieve them. It was close to lunch by this stage, so we stayed for lunch, and the girls had a bit of a play in the back yard while Megan's little sister napped. It was getting close to time to leave for our flu shots, so I decided to just bike directly to the doctor from Megan's place. I realised after we left that we'd still left the stuffed toys behind, but the plan was to drive back after our flu shots and have another swim their neighbour's pool, so it was all good. We got to the doctor, and waited for Sarah to arrive. Sarah and I weren't existing patients at Zoe's doctor, but we'd decided to get the flu shot as a family to try and ease the experience for Zoe. We both had to do new patient intake stuff before we had a consult with Zoe's doctor and got prescriptions for the flu shot. I popped next door to the adjacent pharmacy get the prescriptions filled, and then the nurse gave us the shots. For the last round of vaccinations that Zoe received, she needed three, and she screamed the building down at the first jab. The poor nurse was very shaken, so we've been working to try and get her to feel more at ease about this one. Zoe went first, and she took a deep breath, and she was winding up to freak out when she had her shot, but then it was all over, and she let the breath go, and looked around with a kind of "is that it?" reaction. She didn't even cry. I was so proud of her. I got my shot, and then Sarah got hers, and we had to sit in the waiting room for 10 minutes to make sure we didn't turn into pumpkins, and we were on our way. We biked home, I grabbed our swim gear, and we drove back to Megan's place. The pool ended up being quite cold. Megan didn't want to get in, and Zoe didn't last long either. Megan's Mum was working back late, so I invited Megan, her Dad and her sister over for dinner, and we headed home so I could prepare it. One of Zoe's stuffed toys had been located. We had a nice dinner of deviled sausages made in the Thermomix, and for a change I didn't have a ton of leftovers. Jason had found the other stuffed toy in his truck, so we'd finally tracked them both down. After Megan and family went home, I got Zoe to bed without much fuss, and pretty much on time. I think she should sleep well tonight.

16 April 2014

Wouter Verhelst: Call for help for DVswitch maintenance

I've taken over "maintaining" DVswitch from Ben Hutchings a few years ago, since Ben realized he didn't have the time anymore to work on it well. After a number of years, I have to admit that I haven't done a very good job. Not becase I didn't want to work on it, but mainly because I don't have enough time to fix DVswitch against the numerous moving targets that it uses; the APIs of libav and of liblivemedia are fluent enough that just making sure everything remains compilable and in working order is quite a job. DVswitch is used by many people; DebConf, FOSDEM, and the CCC are just a few examples, but I know of at least three more. Most of these (apart from DebConf and FOSDEM) maintain local patches which I've been wanting to merge into the upstream version of dvswitch. However, my time is limited, and over the past few years I've not been able to get dvswitch into a state where I confidently felt I could upload it into Debian unstable for a release. One step we took in order to get that closer was to remove the liblivemedia dependency (which implied removing the support for RTSP sources). Unfortunately, the resulting situation wasn't good enough yet, since libav had changed API enough that current versions of DVswitch compiled against current versions of libav will segfault if you try to do anything useful. I must admit to myself that I don't have the time and/or skill set to maintain DVswitch on an acceptable level all by myself. So, this is a call for help: If you're using DVswitch for your conference and want to continue doing so, please talk to us. The first things we'll need to do: See you there?

Richard Hartmann: secure password storage

Dear lazyweb, for obvious reaons I am in the process of cycling out a lot of passwords. For the last decade or so, I have been using openssl.vim to store less-frequently-used passwords and it's still working fine. Yet, it requires some manual work, not least of which manually adding random garbage at the start of the plain text (and in other places) every time I save my passwords. In the context of changing a lot of passwords at once, this has started to become tedious. Plus, I am not sure if a tool of the complexity and feature-set of Vim is the best choice for security-critical work on encrypted files. Long story short, I am looking for alternatives. I did some research but couldn't come up with anything I truly liked; as there's bound to be tools which fit the requirements of like-minded people, I decided to ask around a bit. My personal short-list of requirements is: Any and all feedback appreciated. Depending on the level of feedback, I may summarize my own findings and suggestions into a follow-up post.

Andrew Pollock: [life] Day 78: Alginate, dragon boats and relatives

I ordered some alginate the other day, and it arrived yesterday, but we were out, so I had to pick it up from the post office this morning. Anshu and I picked it up before Zoe was dropped off. We had a couple of attempts at making some, but didn't quite get the ratios or the quantity right, and we were too slow, so we'll have to try again. The plan is to try and make a cast of Zoe's hand, since we were messing around with plaster of Paris recently. I've found a good Instructable to try and follow. Nana and her dragon boating team were competing in the Australian Dragon Boat Championships over Easter, and her first race was today. It also ended up that today was the best day to try and go and watch, so when she called to say her first race would be around noon, I quickly decided we should jump in the car and head up to Kawana Waters. We abandoned the alginate, and I slapped together a picnic lunch for Zoe and I, and we bid Anshu farewell and drove up. Zoe's fever seemed to break yesterday afternoon after Sarah picked her up, and she slept well, but despite all that, she napped in the car on the way up, which was highly unusual, but helped pass the time. She woke up when we arrived. I managed to get a car park not too far from the finish line, and we managed to find Nana, whose team was about the enter the marshaling area. Her boat was closest to the shore we were watching from, and her boat came second in their qualifying round for the 200 metre race, meaning they went straight through to the semi-finals. The semi-finals were going to be much later, and I wanted to capitalise on the fact that we were going to have to drive right past my Mum and Dad's place on the way home to try and see my sister and her family, since we missed them on Monday. We headed back after lunch and a little bit of splashing around in the lake, and ended up staying for dinner at Mum and Dad's. Zoe had a great time catching up with her cousin Emma, and fooling around with Grandpa and Uncle Michael. She got to bed a little bit late by the time we got home, but I'm hopeful she'll sleep well tonight.

David Pashley: Bad Password Policies

After the whole Heartbleed fiasco, I ve decided to continue my march towards improving my online security. I d already begun the process of using LastPass to store my passwords and generate random passwords for each site, but I hadn t completed the process, with some sites still using the same passwords, and some having less than ideal strength passwords, so I spent some time today improving my password position. Here s some of the bad examples of password policy I ve discovered today. First up we have Live.com. A maximum of 16 characters from the Microsoft auth service. Seems to accept any character though. Screenshot from 2014-04-15 21:36:57 This excellent example is from creditexpert.co.uk, one of the credit agencies here in the UK. They not only restrict to 20 characters, they restrict you to @, ., _ or . So much for teaching people how to protect themselves online. Screenshot from 2014-04-15 17:38:28 Here s Tesco.com after attempting to change my password to QvHn#9#kDD%cdPAQ4&b&ACb4x%48#b . If you can figure out how this violates their rules, I d love to know. And before you ask, I tried without numbers and that still failed so it can t be the three and only three thing. The only other idea might be that they meant i.e. rather than e.g. , but I didn t test that. Screenshot from 2014-04-15 16:20:17 Edit: Here is a response from Tesco on Twitter: Screenshot from 2014-04-16 07:47:58 Here s a poor choice from ft.com, refusing to accept non-alphanumeric characters. On the plus side they did allow the full 30 characters in the password. Screenshot from 2014-04-15 15:22:08 The finest example of a poor security policy is a company who will remain nameless due to their utter lack of security. Not only did they not use HTTPS, they accepted a 30 character password and silently truncated it to 20 characters. The reason I know this is because when I logged out and tried to log in again and then used the forgot my password option, they emailed me the password in plain text. I have also been setting up two-factor authentication where possible. Most sites use the Google Authenticator application on your mobile to give you a 6 digit code to type in in addition to your password. I highly recommend you set it up too. There s a useful list of sites that implement 2FA and links to their documentation at http://twofactorauth.org/. I realise that my choice LastPass requires me to trust them, but I think the advantages outweigh the disadvantages of having many sites using the same passwords and/or low strength passwords. I know various people cleverer than me have looked into their system and failed to find any obvious flaws. Remember people, when you implement a password, allow the following things: If you are going to place restrictions, please make sure the documentation matches the implementation, provide a client-side implementation to match and provide quick feedback to the user, and make sure you explicitly say what is wrong with the password, rather than referring back to the incorrect documentation. There are also many JS password strength meters available to show how secure the inputted passwords are. They are possibly a better way of providing feedback about security than having arbitrary policies that actually harm your security. As someone said to me on twitter, it s not like password is too strong was ever a bad thing. The post Bad Password Policies appeared first on David Pashley.com.

15 April 2014

Petter Reinholdtsen: FreedomBox milestone - all packages now in Debian Sid

The Freedombox project is working on providing the software and hardware to make it easy for non-technical people to host their data and communication at home, and being able to communicate with their friends and family encrypted and away from prying eyes. It is still going strong, and today a major mile stone was reached. Today, the last of the packages currently used by the project to created the system images were accepted into Debian Unstable. It was the freedombox-setup package, which is used to configure the images during build and on the first boot. Now all one need to get going is the build code from the freedom-maker git repository and packages from Debian. And once the freedombox-setup package enter testing, we can build everything directly from Debian. :) Some key packages used by Freedombox are freedombox-setup, plinth, pagekite, tor, privoxy, owncloud and dnsmasq. There are plans to integrate more packages into the setup. User documentation is maintained on the Debian wiki. Please check out the manual and help us improve it. To test for yourself and create boot images with the FreedomBox setup, run this on a Debian machine using a user with sudo rights to become root:
sudo apt-get install git vmdebootstrap mercurial python-docutils \
  mktorrent extlinux virtualbox qemu-user-static binfmt-support \
  u-boot-tools
git clone http://anonscm.debian.org/git/freedombox/freedom-maker.git \
  freedom-maker
make -C freedom-maker dreamplug-image raspberry-image virtualbox-image
Root access is needed to run debootstrap and mount loopback devices. See the README in the freedom-maker git repo for more details on the build. If you do not want all three images, trim the make line. Note that the virtualbox-image target is not really virtualbox specific. It create a x86 image usable in kvm, qemu, vmware and any other x86 virtual machine environment. You might need the version of vmdebootstrap in Jessie to get the build working, as it include fixes for a race condition with kpartx. If you instead want to install using a Debian CD and the preseed method, boot a Debian Wheezy ISO and use this boot argument to load the preseed values:
url=http://www.reinholdtsen.name/freedombox/preseed-jessie.dat
I have not tested it myself the last few weeks, so I do not know if it still work. If you wonder how to help, one task you could look at is using systemd as the boot system. It will become the default for Linux in Jessie, so we need to make sure it is usable on the Freedombox. I did a simple test a few weeks ago, and noticed dnsmasq failed to start during boot when using systemd. I suspect there are other problems too. :) To detect problems, there is a test suite included, which can be run from the plinth web interface. Give it a go and let us know how it goes on the mailing list, and help us get the new release published. :) Please join us on IRC (#freedombox on irc.debian.org) and the mailing list if you want to help make this vision come true.

B lint R czey: Proposing amd64-hardened architecture for Debian

Facing last week s Heartbleed bug the need for improving the security of our systems became more apparent than usually. In Debian there are widely used methods for Hardening packages at build time and guidelines for improving the default installations security. Employing such methods usually come at an expense, for example slower code execution of binaries due to additional checks or additional configuration steps when setting up a system. Balancing between usability and security Debian chose an approach which would satisfy the most users by using C/C++ features which only slightly decrease execution speed of built binaries and by using reasonable defaults in package installations. All the architectures supported by Debian aims using the same methods for enhancing security but it does not have to stay the same way. Amd64 is the most widely used architecture of Debian according to popcon and amd64 hardware comes with powerful CPU-s. I think there would be a significant amount of people (being one of them :-)) who would happily use a version of Debian with more security features enabled by default sacrificing some CPU power and installing and setting up additional packages. My proposal for serving those security-focused users is introducing a new architecture targeting amd64 hardware, but with more security related C/C++ features turned on for every package (currently hardening has to be enabled by the maintainers in some way) through compiler flags as a start. Introducing the new architecture would also let package maintainers enabling additional dependencies and build rules selectively for the new architecture improving the security further. On the users side the advantage of having a separate security enhanced architecture instead of a Debian derivative is the potential of installing a set of security enhanced packages using multiarch. You could have a fast amd64 installation as a base and run Apache or any other sensitive server from the amd64-hardened packages! I have sent the proposal for discussion to debian-dev, too. Please join the discussion there or leave a comment here. Update: Many of you wondered if amd64-hardened could have prevented the exploitation of the Heartbleed vulnerability. I have posted a proof of concept to show that using -fsanitize=address and disabling custom freelist would have protected systems against stealing data using the exploits. Disabling the custom freelist-like solutions and enabling-fsanitize=address would be part of amd64-hardened to make memory protection techniques work effectively thus I think if we had this architecture ready at the beginning of April, it would have been immune to Heartbleed.

Dirk Eddelbuettel: BH release 1.54.0-2

Yesterday's release of RcppBDT 0.2.3 lead to an odd build error. If one used at the same time a 32-bit OS, a compiler as recent as g++ 4.7 and the Boost 1.54.0 headers (directly or via the BH package) then the file lexical_cast.hpp barked and failed to compile for lack of an 128-bit integer (which is not a surprise on a 32-bit OS). After looking at this for a bit, and looking at some related bug report, I came up with a simple fix (which I mentioned in an update to the RcppBDT 0.2.3 release post). Sleeping over it, and comparing to the Boost 1.55 file, showed that the hunch was right, and I have since made a new release 1.54.0-2 of the BH package which contains the fix.
Changes in version 1.54.0-2 (2014-04-14)
  • Bug fix to lexical_cast.hpp which now uses the test for INT128 which the rest of Boost uses, consistent with Boost 1.55 too.
Courtesy of CRANberries, there is also a diffstat report for the most recent release. Comments and suggestions are welcome via the mailing list or issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Colin Watson: Porting GHC: A Tale of Two Architectures

We had some requests to get GHC (the Glasgow Haskell Compiler) up and running on two new Ubuntu architectures: arm64, added in 13.10, and ppc64el, added in 14.04. This has been something of a saga, and has involved rather more late-night hacking than is probably good for me. Book the First: Recalled to a life of strange build systems You might not know it from the sheer bulk of uploads I do sometimes, but I actually don't speak a word of Haskell and it's not very high up my list of things to learn. But I am a pretty experienced build engineer, and I enjoy porting things to new architectures: I'm firmly of the belief that breadth of architecture support is a good way to shake out certain categories of issues in code, that it's worth doing aggressively across an entire distribution, and that, even if you don't think you need something now, new requirements have a habit of coming along when you least expect them and you might as well be prepared in advance. Furthermore, it annoys me when we have excessive noise in our build failure and proposed-migration output and I often put bits and pieces of spare time into gardening miscellaneous problems there, and at one point there was a lot of Haskell stuff on the list and it got a bit annoying to have to keep sending patches rather than just fixing things myself, and ... well, I ended up as probably the only non-Haskell-programmer on the Debian Haskell team and found myself fixing problems there in my free time. Life is a bit weird sometimes. Bootstrapping packages on a new architecture is a bit of a black art that only a fairly small number of relatively bitter and twisted people know very much about. Doing it in Ubuntu is specifically painful because we've always forbidden direct binary uploads: all binaries have to come from a build daemon. Compilers in particular often tend to be written in the language they compile, and it's not uncommon for them to build-depend on themselves: that is, you need a previous version of the compiler to build the compiler, stretching back to the dawn of time where somebody put things together with a big magnet or something. So how do you get started on a new architecture? Well, what we do in this case is we construct a binary somehow (usually involving cross-compilation) and insert it as a build-dependency for a proper build in Launchpad. The ability to do this is restricted to a small group of Canonical employees, partly because it's very easy to make mistakes and partly because things like the classic "Reflections on Trusting Trust" are in the backs of our minds somewhere. We have an iron rule for our own sanity that the injected build-dependencies must themselves have been built from the unmodified source package in Ubuntu, although there can be source modifications further back in the chain. Fortunately, we don't need to do this very often, but it does mean that as somebody who can do it I feel an obligation to try and unblock other people where I can. As far as constructing those build-dependencies goes, sometimes we look for binaries built by other distributions (particularly Debian), and that's pretty straightforward. In this case, though, these two architectures are pretty new and the Debian ports are only just getting going, and as far as I can tell none of the other distributions with active arm64 or ppc64el ports (or trivial name variants) has got as far as porting GHC yet. Well, OK. This was somewhere around the Christmas holidays and I had some time. Muggins here cracks his knuckles and decides to have a go at bootstrapping it from scratch. It can't be that hard, right? Not to mention that it was a blocker for over 600 entries on that build failure list I mentioned, which is definitely enough to make me sit up and take notice; we'd even had the odd customer request for it. Several attempts later and I was starting to doubt my sanity, not least for trying in the first place. We ship GHC 7.6, and upgrading to 7.8 is not a project I'd like to tackle until the much more experienced Haskell folks in Debian have switched to it in unstable. The porting documentation for 7.6 has bitrotted more or less beyond usability, and the corresponding documentation for 7.8 really isn't backportable to 7.6. I tried building 7.8 for ppc64el anyway, picking that on the basis that we had quicker hardware for it and didn't seem likely to be particularly more arduous than arm64 (ho ho), and I even got to the point of having a cross-built stage2 compiler (stage1, in the cross-building case, is a GHC binary that runs on your starting architecture and generates code for your target architecture) that I could copy over to a ppc64el box and try to use as the base for a fully-native build, but it segfaulted incomprehensibly just after spawning any child process. Compilers tend to do rather a lot, especially when they're built to use GCC to generate object code, so this was a pretty serious problem, and it resisted analysis. I poked at it for a while but didn't get anywhere, and I had other things to do so declared it a write-off and gave up. Book the Second: The golden thread of progress In March, another mailing list conversation prodded me into finding a blog entry by Karel Gardas on building GHC for arm64. This was enough to be worth another look, and indeed it turned out that (with some help from Karel in private mail) I was able to cross-build a compiler that actually worked and could be used to run a fully-native build that also worked. Of course this was 7.8, since as I mentioned cross-building 7.6 is unrealistically difficult unless you're considerably more of an expert on GHC's labyrinthine build system than I am. OK, no problem, right? Getting a GHC at all is the hard bit, and 7.8 must be at least as capable as 7.6, so it should be able to build 7.6 easily enough ... Not so much. What I'd missed here was that compiler engineers generally only care very much about building the compiler with older versions of itself, and if the language in question has any kind of deprecation cycle then the compiler itself is likely to be behind on various things compared to more typical code since it has to be buildable with older versions. This means that the removal of some deprecated interfaces from 7.8 posed a problem, as did some changes in certain primops that had gained an associated compatibility layer in 7.8 but nobody had gone back to put the corresponding compatibility layer into 7.6. GHC supports running Haskell code through the C preprocessor, and there's a __GLASGOW_HASKELL__ definition with the compiler's version number, so this was just a slog tracking down changes in git and adding #ifdef-guarded code that coped with the newer compiler (remembering that stage1 will be built with 7.8 and stage2 with stage1, i.e. 7.6, from the same source tree). More inscrutably, GHC has its own packaging system called Cabal which is also used by the compiler build process to determine which subpackages to build and how to link them against each other, and some crucial subpackages weren't being built: it looked like it was stuck on picking versions from "stage0" (i.e. the initial compiler used as an input to the whole process) when it should have been building its own. Eventually I figured out that this was because GHC's use of its packaging system hadn't anticipated this case, and was selecting the higher version of the ghc package itself from stage0 rather than the version it was about to build for itself, and thus never actually tried to build most of the compiler. Editing ghc_stage1_DEPS in ghc/stage1/package-data.mk after its initial generation sorted this out. One late night building round and round in circles for a while until I had something stable, and a Debian source upload to add basic support for the architecture name (and other changes which were a bit over the top in retrospect: I didn't need to touch the embedded copy of libffi, as we build with the system one), and I was able to feed this all into Launchpad and watch the builders munch away very satisfyingly at the Haskell library stack for a while. This was all interesting, and finally all that work was actually paying off in terms of getting to watch a slew of several hundred build failures vanish from arm64 (the final count was something like 640, I think). The fly in the ointment was that ppc64el was still blocked, as the problem there wasn't building 7.6, it was getting a working 7.8. But now I really did have other much more urgent things to do, so I figured I just wouldn't get to this by release time and stuck it on the figurative shelf. Book the Third: The track of a bug Then, last Friday, I cleared out my urgent pile and thought I'd have another quick look. (I get a bit obsessive about things like this that smell of "interesting intellectual puzzle".) slyfox on the #ghc IRC channel gave me some general debugging advice and, particularly usefully, a reduced example program that I could use to debug just the process-spawning problem without having to wade through noise from running the rest of the compiler. I reproduced the same problem there, and then found that the program crashed earlier (in stg_ap_0_fast, part of the run-time system) if I compiled it with +RTS -Da -RTS. I nailed it down to a small enough region of assembly that I could see all of the assembly, the source code, and an intermediate representation or two from the compiler, and then started meditating on what makes ppc64el special. You see, the vast majority of porting bugs come down to what I might call gross properties of the architecture. You have things like whether it's 32-bit or 64-bit, big-endian or little-endian, whether char is signed or unsigned, that sort of thing. There's a big table on the Debian wiki that handily summarises most of the important ones. Sometimes you have to deal with distribution-specific things like whether GL or GLES is used; often, especially for new variants of existing architectures, you have to cope with foolish configure scripts that think they can guess certain things from the architecture name and get it wrong (assuming that powerpc* means big-endian, for instance). We often have to update config.guess and config.sub, and on ppc64el we have the additional hassle of updating libtool macros too. But I've done a lot of this stuff and I'd accounted for everything I could think of. ppc64el is actually a lot like amd64 in terms of many of these porting-relevant properties, and not even that far off arm64 which I'd just successfully ported GHC to, so I couldn't be dealing with anything particularly obvious. There was some hand-written assembly which certainly could have been problematic, but I'd carefully checked that this wasn't being used by the "unregisterised" (no specialised machine dependencies, so relatively easy to port but not well-optimised) build I was using. A problem around spawning processes suggested a problem with SIGCHLD handling, but I ruled that out by slowing down the first child process that it spawned and using strace to confirm that SIGSEGV was the first signal received. What on earth was the problem? From some painstaking gdb work, one thing I eventually noticed was that stg_ap_0_fast's local stack appeared to be being corrupted by a function call, specifically a call to the colourfully-named debugBelch. Now, when IBM's toolchain engineers were putting together ppc64el based on ppc64, they took the opportunity to fix a number of problems with their ABI: there's an OpenJDK bug with a handy list of references. One of the things I noticed there was that there were some stack allocation optimisations in the new ABI, which affected functions that don't call any vararg functions and don't call any functions that take enough parameters that some of them have to be passed on the stack rather than in registers. debugBelch takes varargs: hmm. Now, the calling code isn't quite in C as such, but in a related dialect called "Cmm", a variant of C-- (yes, minus), that GHC uses to help bridge the gap between the functional world and its code generation, and which is compiled down to C by GHC. When importing C functions into Cmm, GHC generates prototypes for them, but it doesn't do enough parsing to work out the true prototype; instead, they all just get something like extern StgFunPtr f(void);. In most architectures you can get away with this, because the arguments get passed in the usual calling convention anyway and it all works out, but on ppc64el this means that the caller doesn't generate enough stack space and then the callee tries to save its varargs onto the stack in an area that in fact belongs to the caller, and suddenly everything goes south. Things were starting to make sense. Now, debugBelch is only used in optional debugging code; but runInteractiveProcess (the function associated with the initial round of failures) takes no fewer than twelve arguments, plenty to force some of them onto the stack. I poked around the GCC patch for this ABI change a bit and determined that it only optimised away the stack allocation if it had a full prototype for all the callees, so I guessed that changing those prototypes to extern StgFunPtr f(); might work: it's still technically wrong, not least because omitting the parameter list is an obsolescent feature in C11, but it's at least just omitting information about the parameter list rather than actively lying about it. I tweaked that and ran the cross-build from scratch again. Lo and behold, suddenly I had a working compiler, and I could go through the same build-7.6-using-7.8 procedure as with arm64, much more quickly this time now that I knew what I was doing. One upstream bug, one Debian upload, and several bootstrapping builds later, and GHC was up and running on another architecture in Launchpad. Success! Epilogue There's still more to do. I gather there may be a Google Summer of Code project in Linaro to write proper native code generation for GHC on arm64: this would make things a good deal faster, but also enable GHCi (the interpreter) and Template Haskell, and thus clear quite a few more build failures. Since there's already native code generation for ppc64 in GHC, getting it going for ppc64el would probably only be a couple of days' work at this point. But these are niceties by comparison, and I'm more than happy with what I got working for 14.04. The upshot of all of this is that I may be the first non-Haskell-programmer to ever port GHC to two entirely new architectures. I'm not sure if I gain much from that personally aside from a lot of lost sleep and being considered extremely strange. It has, however, been by far the most challenging set of packages I've ported, and a fascinating trip through some odd corners of build systems and undefined behaviour that I don't normally need to touch.

14 April 2014

Richard Hartmann: git-annex corner case: Changing commit messages retroactively and after syncing

This is half a blog post and half a reminder for my future self. So let's say you used the following commands:
git add foo
git annex add bar
git annex sync
# move to different location with different remotes available
git add quux
git annex add quuux
git annex sync

what I wanted to happen was to simply sync the already committed stuff to the other remotes. What happened instead was git annex sync's automagic commit feature (which you can not disable, it seems) doing its job: Commit what was added earlier and use "git-annex automatic sync" as commit message. This is not a problem in and as of itself, but as this is my my master annex and as I managed to maintain clean commit messages for the last few years, I felt the need to clean this mess up. Changing old commit messages is easy:
git rebase --interactive HEAD~3

pick the r option for "reword" and amend the two commit messages. I did the same on my remote and all the branches I could find with git branch -a. Problem is, git-annex pulls in changes from refs which are not shown as branches; run git annex sync and back are the old commits along with a merge commit like an ugly cherry on top. Blegh. I decided to leave my comfort zone and ended up with the following:
# always back up before poking refs
git clone --mirror repo backup
git reset --hard 1234
git show-ref   grep master
# for every ref returned, do:
  git update-ref $ref 1234

rinse repeat for every remote, git annex sync, et voil . And yes, I avoided using an actual loop on purpose; sometimes, doing things slowly and by hand just feels safer. For good measure, I am running
git fsck && git annex fsck

on all my remotes now, but everything looks good up to now.

Daniel Kahn Gillmor: OTR key replacement (heartbleed)

I'm replacing my OTR key for XMPP because of heartbleed (see below). If the plain ASCII text below is mangled beyond verification, you can retrieve a copy of it from my web site that should be able to be verified.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
OTR Key Replacement for XMPP dkg@jabber.org
===========================================
Date: 2014-04-14
My main XMPP account is dkg@jabber.org.
I prefer OTR [0] conversations when using XMPP for private
discussions.
I was using irssi to connect to XMPP servers, and irssi relies on
OpenSSL for the TLS connections.  I was using it with versions of
OpenSSL that were vulnerable to the "Heartbleed" attack [1].  It's
possible that my OTR long-term secret key was leaked via this attack.
As a result, I'm changing my OTR key for this account.
The new, correct OTR fingerprint for the XMPP account at dkg@jabber.org is:
  F8953C5D 48ABABA2 F48EE99C D6550A78 A91EF63D
Thanks for taking the time to verify your peers' fingerprints.  Secure
communication is important not only to protect yourself, but also to
protect your friends, their friends and so on.
Happy Hacking,
  --dkg  (Daniel Kahn Gillmor)
Notes:
[0] OTR: https://otr.cypherpunks.ca/
[1] Heartbleed: http://heartbleed.com/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQJ8BAEBCgBmBQJTTBF+XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w
ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXRFQjk2OTEyODdBN0FEREUzNzU3RDkxMUVB
NTI0MDFCMTFCRkRGQTVDAAoJEKUkAbEb/fpcYwkQAKLzEnTV1lrK6YrhdvRnuYnh
Bh9Ad2ZY44RQmN+STMEnCJ4OWbn5qx/NrziNVUZN6JddrEvYUOxME6K0mGHdY2KR
yjLYudsBuSMZQ+5crZkE8rjBL8vDj8Dbn3mHyT8bAbB9cmASESeQMu96vni15ePd
2sB7iBofee9YAoiewI+xRvjo2aRX8nbFSykoIusgnYG2qwo2qPaBVOjmoBPB5YRI
PkN0/hAh11Ky0qQ/GUROytp/BMJXZx2rea2xHs0mplZLqJrX400u1Bawllgz3gfV
qQKKNc3st6iHf3F6p6Z0db9NRq+AJ24fTJNcQ+t07vMZHCWM+hTelofvDyBhqG/r
l8e4gdSh/zWTR/7TR3ZYLCiZzU0uYNd0rE3CcxDbnGTUS1ZxooykWBNIPJMl1DUE
zzcrQleLS5tna1b9la3rJWtFIATyO4dvUXXa9wU3c3+Wr60cSXbsK5OCct2KmiWY
fJme0bpM5m1j7B8QwLzKqy/+YgOOJ05QDVbBZwJn1B7rvUYmb968yLQUqO5Q87L4
GvPB1yY+2bLLF2oFMJJzFmhKuAflslRXyKcAhTmtKZY+hUpxoWuVa1qLU3bQCUSE
MlC4Hv6vaq14BEYLeopoSb7THsIcUdRjho+WEKPkryj6aVZM5WnIGIS/4QtYvWpk
3UsXFdVZGfE9rfCOLf0F
=BGa1
-----END PGP SIGNATURE-----

Christine Spang: PyCon 2014 retrospective

PyCon 2014 happened. (Sprints are still happening.) This was my 3rd PyCon, but my first year as a serious contributor to the event, which led to an incredibly different feel. I also came as a person running a company building a complex system in Python, and I loved having the overarching mission of what I'm building driving my approach to what I chose to do. PyCon is one of the few conferences I go to where the feeling of acceptance and at-homeness mitigates the introvert overwhelm at nonstop social interaction. It's truly a special event and community. Here are some highlights: I didn't get to go to a lot of talks in person this year since my personal schedule was so full, but the PyCon video team is amazing as usual, so I'm looking forward to checking out the archive. It really is a gift to get the videos up while energy from the conference is still so high and people want to check out things they missed and share the talks they loved. Thanks to everyone, hugs, peace out, et cetera!

Next.

Previous.