Search Results: "xam"

19 April 2015

Laura Arjona: Six months selfhosting: my userop experiences

Note: In this post I mention some problems and ask questions (to myself, like thinking aloud ). The goal is not to get answers to those questions (I suppose that I will find them soon or later in the internet, manuals and so), but to show the kind of problems and questions that arise in my selfhosting adventures, which I suppose are common to other people trying to administer a home server with some web services. Am I an userop? Well I m something in the middle of (GNU/Linux) user and sysadmin: I have studied computer technical engineering but most of my experience has been in helpdesk, providing support for Windows users. I m running Debian in some LAMP boxes at work (without GUI) since 2008 or so, and in my desktops (with GUI) since 2010. I don t code nor package, but I don t mind trying to read code and understand it (or not). I know a bit of C, a bit of Python, of PHP, and enough Perl to open a Perl file and close it after two minutes, understanding that it s great, but too much for me :) I translate software, so I m not scared to clone a repository, edit files, commit or submit a patch. I m not scared of compiling a program (except if it s an Android app: I try to avoid setting up the development environment just to try some translation that I made but I built my Puma before it was the binary available for download or in F-Droid). In conclusion, I feel more like a GNU/Linux power user than a sysadmin . Sometimes just a user or even a newbie (for example, I don t know very well the Unix/Linux folder tree where are the wallpapers stored? Does it depend on the desktop that I use?). Anyway. I won t stop my free software + free networks digital life because I don t know many things. I bought a small server for home last September, and I wanted to try to selfhost some services, for me and for my family. I want to be a home sysadmin or something like that, so I joined the userops mailing list :) Here you have my experiences on selfhosting/being an userop until now. Mail I even didn t try to setup my mail server, because many people say it s a pain (although nice articles were published about how to do it, for example this series in ArsTechnica) and I need a static IP which is 14 /month more to my ISP, and Gandi, the place where I rented my domain name, provides mail, and they use Debian and Roundcube, and sponsor Debian too, so I decided to trust on them. So this is my strategy now, to try to keep mail under my control: MediaGoblin I ve setup two MediaGoblin instances (yes, two!). I managed to do it in Debian 7 stable (I think NodeJS npm was not needed then), but soon later I upgraded to Jessie so now it s even better. I installed Nginx and PostgreSQL via apt, to use them for both instances (and probably some more software later). One instance is public, and I use a Debian user, a PostgreSQL database, and it s running in http://media.larjona.net
I have requested an SSL cert to Gandi but I still didn t deployed it (lazy LArjona!!). The other instance is private, for family photos. I didn t know very well how much of my existing setup could reuse and how to keep both instances in case of downtimes or attack I know more or less the concept or chroot but I don t know how to deploy it in my machine. So I decided to use another Debian user, another PostgreSQL database, deploy MediaGoblin in a different folder, and create another virtual server in my Nginx to serve it. I managed to setup that virtual server to http-authenticate and to serve content via a different port, and use a self-signed SSL certificate (it s only for family, so it does not matter). I created another (unprivileged) Debian user with a password for the nginx authentication, and gave my family the URL in the form https://mediaprivate.larjona.net:PortNumber and the user and password (mediaprivate is a string, and PortNumber is a number). I think they don t use the instance too much, but at least I upload photos there from time to time and email the link instead of emailing the photos themselves (they don t use GPG either ). Upgrades I upgraded MediaGoblin from 0.7.1 to 0.8.0 successfully, I sent a report about how I did it to the mailing list. First I upgraded the public instance, when I figure out the process, I upgraded the second instance to test my instructions, and then, I sent the report with the instructions to the mailing list. Static site and LimeSurvey: the power of free software (with instructions) I wanted to act as a mirror of floss2013.libresoft.es and surveys.libresoft.es since they suffer a downtime and I participated in that project (not in the sysadmin part, but in the research and content creation). The static site floss2013.libresoft.es offered a zip with the whole website tree (since the website was licensed as AGPL), and I had access to the git repo holding the development copy of the website. So I just cloned the repository and setup another nginx virtual server in my machine, and tuned my DNS zone in Gandi website to serve floss2013.larjona.net from home. 10 minutes setup YAY! #inGitWeTrust #FreeSoftwareFTW :) For surveys.larjona.net I had to install a LimeSurvey instance. I knew how to do it because we use LimeSurvey at work, but at home I had Nginx instead of Apache, and PostgreSQL instead of MySQL. And no PHP I searched about how to install PHP in Nginx (I can use apt-get, nice!) and how to install LimeSurvey with Nginx and PostgreSQL (I had documentation about that, so I followed, and it worked). For making available the data (one survey and its results, so people can login as visitor to query and get statistics), I downloaded the LimeSurvey export dataset that we were providing in the static website, followed the replication instructions (hey, I wrote them!), and they worked #oleole! (And here, dear researchers, gets demonstrated that free software and free culture really empower your research and help spreading your results). Etherpad: not so easy, it seems! I m trying to install Etherpad-Lite, but I m suffering a bit. I think I did everything ok according to some guides but I get Bad Gateway and these kind of errors when trying to browse with Lynx in the host:
[error] 3615#0: *24 upstream timed out (110: Connection timed out) 
while reading response header from upstream, 
client: 127.0.0.1, 
server: pad.larjona.net, 
request: "GET / HTTP/1.0", 
upstream: "http://127.0.0.1:9001/", 
host: "pad.larjona.net"
2015/04/17 20:52:56 [error] 3615#0: *24 connect() failed 
(111: Connection refused) while connecting to upstream, 
client: 127.0.0.1, 
server: pad.larjona.net, 
request: "GET / HTTP/1.0", 
upstream: "http://[::1]:9001/", 
host: "pad.larjona.net"
I m not sure if I need to open some port in iptables, my router, or change my nginx configuration because the guides assume you re only serving one website in the port 80 (and I have several of them, now ), or what I ve spent three chunks of time (maybe ~2h each?) on this, in different days, and couldn t figure it out, so I decided to round-robin in my TODO list. Userops thoughts Debian brings peace of mind (for me) On one side, maintaining a Debian box it s quite easy, and the more software that it s packaged, the less time that I spend installing or upgrading. I like being in stable, I m in Jessie now (I migrated when it was frozen), but I ll stay in stable as much as I can. I like that I can use the software that I installed via apt-get for several services (nginx, PostgreSQL ). About the software that is not packaged (MediaGoblin, LimeSurvey, EtherPad, maybe others later), I wonder how dependencies and updates are handled. And maybe (probably) I have installed some components several times, one for each service (this sounds like a Windows box #grr). For example MediaGoblin uses PyPump. PyPump 0.5 is packaged in Debian Jessie. MediaGoblin uses PyPump 0.7+. What if PyPump 0.7+ gets, let s say, into Jessie-backports? Can I benefit from that? I know that MediaGoblin upgrade instructions includes upgrading the dependencies, but what about some security patch in one dependency? Should I upgrade the pip modules periodically? How to know if some upgrade is recommended because patches a vulnerability, or it s just new features (and maybe breaking my setup)? This kind of things are the peace of mind that Debian packaging brings to me: when some piece of software is packaged, I know maybe I need to care about proper setup and configuration, but later, it s kind-of-easy to maintain (since the Debian maintainers care about the rest). I don t mind about cloning a repo and compiling, I mind about later, or coexistance with other program/services. I trust in the MediaGoblin community and I m an active member (I m not developer, but hang on IRC, follow the mailing list, etc) but for example I don t know anything about the EtherPad project. And I don t feel like joining the community (I m already an active member in Debian, MediaGoblin, F-Droid, Pump.io, translator of LimeSurvey and many other small apps that I use, and in the future will use more services, like OwnCloud, XMPP ), joining the community of each software that I use is becoming not sustainable :s Free software is more than software I follow the userop mailing list, and it s becoming very technical. I mostly understand the problems (which are similar to the problems that I face: how to isolate different services, how to easily-configure them, how to make them installable by average user ) But I don t understand most of the solutions proposed, and I think that probably we need technical solutions, but in the meanwhile, some issues can be addressed not with software, but with other means: good documentation, community support, translations, beta-testers This is my conclusion until now. When a project is well documented, I think I can find my way to selfhost no matter if the software is packaged (or contained ) or not. MediaGoblin, and LimeSurvey, are well documented, and the user support channels are very responsive. I find lots of instructions that assume that you will use a whole machine for their service (and not for other things). And lots of documentation for the LAMP stack, but not for Nginx + PostgreSQL and Node instead of PHP So, for each particularity of my setup, I search the internet and try to pick good sources to help me do what I wanted to do. I m kind of privileged Some elements, not software related, to take into account as pre-requisites for succeed selfhosting services: These aspects are not present in a lot of people. If I look around to the computer users that I know (mostly Windows+Android, some GNU/Linux users, some Mac OSX users, some iOS users), I find that they search things like X does not work or they cannot write a proper search query in English. Or they trust some random person writing a recipe in their blog, without trying first to understand what the recipe does. Other people just say I m not a professional sysadmin, I ll just do what everybody does (aka use Google services or whatever). What if I try and I don t succeed? . Things like that. We may need some technical solutions (and hackers are thinking about that, and working on that). But I feel that we need (more) a huge group of beta-testers, dogfooding people, aventurers that try the half-cooked solutions and provide successful and unsuccessful experiences, to guide the research and make software technologies advance. I m not sure if I am an userop, but I feel part of that vanguard force , I want to be part of the future of free software and free networks. Comments? You can comment about this post on this pump.io thread.
Filed under: My experiences and opinion Tagged: Communities, Contributing to libre software, Debian, Developer motivations, English, free networks, Free Software, Freedom, innovation, MediaGoblin, Moving into free software, Project Management, selfhosting, sysadmin

17 April 2015

EvolvisForge blog: Tricks for using Googlemail at work

For these who similarily suffer from having to use Googlemail at work. If anyone else has more of these, please do share. Deactivate the spamfilter The site admins can do that. Otherwise, you will have work-relevant eMails, for example from your own OTRS system, end up in Spam (where you don t see it, as their IMAP sucks) and deleted without asking 30 days later. (AIUI, the only way to get eMails actually deleted from Google ) Do not use their SMTP service Use your own outgoing MTA. This brings back the, well, not feature but should-have-been-granted-but-Google-doesn t-do-it-anyway that, when you write to a mailing list, you also get your own messages into your own INBOX. Calendars I have no solutions for this. I stopped using the Googlemail calendars because they didn t think it a problem that, when I accept an invitation in Kontact (KDEPIM as packaged in Debian sid), the organiser of the calendar item in the sender s calendar (for which I do not have write permissions) changes to me (so the actual meeting organiser cannot change anything afterwards) and/or calendar items get doubled. I now run a local uw-imapd (forward-ported to sid by means of a binNMU) for sent-mail folders etc. and a local iCalendar directory for calendars.

15 April 2015

Joachim Breitner: Talk and Article on Monads for Reverse Engineering

In a recent project of mine, a tool to analyze and create files for the Ravensburger Tiptoi pen, I used two interesting Monads with good results: As that s quite neat, I write a blog post for the German blog funktionale-programmierung.de about it, and also held a talk at the Karlsruhe functional programmers group. If you know some German, enjoy; if not, wait until I have a reason to hold the talk in English. (As a matter of fact, I did hold the talk in English, but only spontaneously, so the text is in German only so far.)

Petter Reinholdtsen: Debian Edu interview: Shirish Agarwal

It was a surprise to me to learn that project to create a complete computer system for schools I've involved in, Debian Edu / Skolelinux, was being used in India. But apparently it is, and I managed to get an interview with one of the friends of the project there, Shirish Agarwal. Who are you, and how do you spend your days? My name is Shirish Agarwal. Based out of the educational and historical city of Pune, from the western state of Maharashtra, India. My bread comes from giving training, giving policy tips, installations on free software to mom and pop shops in different fields from Desktop publishing to retail shops as well as work with few software start-ups as well. How did you get in contact with the Skolelinux / Debian Edu project? It started innocently enough. I have been using Debian for a few years and in one local minidebconf / debutsav I was asked if there was anything for schools or education. I had worked / played with free educational softwares such as Gcompris and Stellarium for my many nieces and nephews so researched and found Debian Edu or Skolelinux as it was known then. Since then I have started using the various education meta-packages provided by the project. What do you see as the advantages of Skolelinux / Debian Edu? It's closest I have seen where a package full of educational software are packed, which are free and open (both literally and figuratively). Even if I take the simplest software which is gcompris, the number of activities therein are amazing. Another one of the softwares that I have liked for a long time is stellarium. Even pysycache is cool except for couple of issues I encountered #781841 and #781842. I prefer software installed on the system over web based solutions, as a web site can disappear any time but the software on disk has the possibility of a larger life span. Of course with both it's more a question if it has enough users who make it fun or sustainable or both for the developer per-se. What do you see as the disadvantages of Skolelinux / Debian Edu? I do see that the Debian Edu team seems to be short-handed and I think more efforts should be made to make it popular and ask and take help from people and the larger community wherever possible. I don't see any disadvantage to use Skolelinux apart from the fact that most apps. are generic which is good or bad how you see it. However, saying that I do acknowledge the fact that the canvas is pretty big and there are lot of interesting ideas that could be done but for reasons not known not done or if done I don't know about them. Let me share some of the ideas (these are more upstream based but still) I have had for a long time : 1. Classical maths question of two trains in opposing directions each running @x kmph/mph at y distance, when they will meet and how far would each travel and similar questions like these. The computer is a fantastic system where questions like these can be drawn, animated and the methodology and answers teased out in interactive manner. While sites such as the Ask Dr. Math FAQ on The Two Trains problem (as an example or point of inspiration) can be used there is lot more that can be done. I dunno if there is a free software which does something like this. The idea being a blend of objects + animation + interaction which does this. The whole interaction could be gamified with points or sounds or colourful celebration whenever the user gets even part of the question or/and methodology right. That would help reinforce good behaviour. This understanding could be used to share/showcase everything from how the first wheel came to be, to evolution to how astronomy started, psychics and everything in-between. One specific idea in the train part was having the Linux mascot on one train and the BSD or GNU mascot on the other train and they meeting somewhere in-between. Characters from blender movies could also be used. 2. Loads of crossword-puzzles with reference to subjects: We have enormous data sets in Wikipedia and Wikitionary. I don't think it should be a big job to design crossword puzzles. Using categories and sub-categories it should be doable to have Q&A single word answers from the existing data-sets. What would make it easy or hard could be the length of the word + existence of many or few vowels depending on the user's input. 3. Jigsaw puzzles - We already have a great software called palapeli with number of slicers making it pretty interesting. What needs to be done is to download large number of public domain and copyleft images, tease and use IPTC tags to categorise them into nature, history etc. and let it loose. This could turn to be really huge collection of images. One source could be taken from commons.wikimedia.org, others could be huge collection of royalty-free stock photos. Potential is immense. Apart from this, free software suffers in two directions, we lag both in development (of using new features per-se) and maintenance a lot. This is more so in educational software as these applications need to be timely and the opportunity cost of missing deadlines is immense. If we are able to solve issues of funding for development and maintenance of such software I don't see any big difficulties. I know of few start-ups in and around India who would love to develop and maintain such software if funding issues could be solved. Which free software do you use daily? That would be huge list. Some of the softwares are obviously apt, aptitude, debdelta, leafpad, the shell of course (zsh nowadays), quassel for IRC. In games I use shisen-sho while card-games are evenly between kpat and Aiselriot. In desktops it's a tie between gnome-flashback and mate. Which strategy do you believe is the right one to use to get schools to use free software? I think it should first start with using specific FOSS apps. in whatever environment they are. If it's MS-Windows or Mac so be it. Once they are habitual with the apps. and there is buy-in from the school management then it could be installed anywhere. Most of the people now understand the concept of a repository because of the various online stores so it isn't hard to convince on that front. What is harder is having enough people with technical skills and passion to service them. If you get buy-in from one or two teachers then ideas like above could also be asked to be done as a project as well. I think where we fall short more than anything is in marketing. For instance, Debian has this whole range of fonts in its archive but there isn't even a page where all those different fonts in the La Ipsum format could be tried out for newcomers. One of the issues faced constantly in installations is with updates and upgrades. People have this myth that each update and upgrade means the user interface will / has to change. I have seen this innumerable times. That perhaps is one of the reasons which browsers like Iceweasel / Firefox change user interfaces so much, not because it might be needed or be functional but because people believe that changed user interfaces are better. This, can easily be pointed with the user interfaces changed with almost every MS-Windows and Mac OS releases. The problems with Debian Edu for deployment are many. The biggest is the huge gap between what is taught in schools and what Debian Edu is aimed at. Me and my friends did teach on week-ends in a government school for around 2 years, and gathered some experience there. Some of the things we learnt/discovered there was :
  1. Most of the teachers are very territorial about their subjects and they do not want you to teach anything out of the portion/syllabus given.
  2. They want any activity on the system in accordance to whatever is in the syllabus.
  3. There are huge barriers both with the English language and at times with objects or whatever. An example, let's say in gcompris you have objects falling down and you have to name them and let's say the falling object is a hat or a fedora hat, this would not be as recognizable as say a Puneri Pagdi so there is need to inject local objects, words wherever possible. Especially for word-games there are so many hindi words which have become part of english vocabulary (for instance in parley), those could be made into a hinglish collection or something but that is something for upstream to do.

14 April 2015

Neil Williams: Extending an existing ARMMP initramfs

The actual use of this extension is still in development and the log files are not currently publicly visible, but it may still be useful for people to know the what and why The Debian ARMMP kernel can be used for multiple devices, just changing the DTB. I ve already done tests with this for Cubietruck and Beaglebone-Black, iMX.53 was one of the original test devices too. Whilst these tests can deploy a full image (there are examples of building such images in the vmdebootstrap package), it is a lot quicker to do simple tests of a kernel using a ramdisk. The default Debian initramfs has a focused objective but still forms a useful base for extension. In particular, I want to be able to test one initramfs on multiple boards (so multiple dtbs) with the same kernel image. I then want to be able, on selected boards, to mount a SATA drive or write an image to eMMC or a USB stick or whatever. LAVA (via the ongoing refactoring, not necessarily in the current dispatcher code) can automate such tests, e.g. to allow me to boot a Cubietruck into a standard Debian ARMMP armhf install on the SATA drive but using a modified (or updated) ARMMP kernel over TFTP without needing to install it on the device itself. That same kernel image can then be tested on multiple boards to see if the changes have benefitted one board at the expense of another. Automating all of that could be of a lot of benefit to the ARM kernel developers in Debian and outside Debian. So, the start point. Install Debian onto a Cubietruck in my case, with a SATA drive attached. All well and good so far, standard Debian Jessie ARMMP. (Cubietruck uses the LPAE kernel flavour but that won t matter for the initramfs.) Rather than building the initramfs manually, this provides a shortcut at some point I may investigate how to do this in QEMU but for now, it s just as quick to SSH onto the Cubietruck and update. I ve already written a little script to download the relevant linux-image package for ARMMP, unpack it and pull out the vmlinuz, the dtbs and a selected list of modules. The list is selective because TFTP has a 32Mb download limit and there are more modules than that. So I borrowed a snippet from the Xen folks (already shown previously here). The script is in a support repository for LAVA but can be used anywhere. (You ll need to edit the package name in the script to choose between ARMMP and ARMMP LPAE. Steps
  1. Get a working initramfs from an installed device running Debian ARMMP and copy some files for use later. Note: use the name of the symlink in the copy so that the file in /tmp/ is the actual file, using the name of the symlink as the filename. This is important later as it saves a step of having to make the (unnecessary) symlink inside the initramfs. Also, mkinitramfs, which built this initrd.img file in the first place, uses the same shared libraries as the main system, so copying these into the initramfs still works. (This is really useful when you get your ramdisk to support the attached secondary storage, allowing you to simply mount the original Debian install and fixup the initramfs by copying files off the main Debian install.) The relevant files are to support DNS lookup inside the initramfs which then allows a test to download a new image to put onto the attached media before rebooting into it.
    cp /boot/initrd.img-3.16.0-4-armmp-lpae /tmp/
    cp /lib/arm-linux-gnueabihf/libresolv.so.2 /tmp/
    cp /mnt/lib/arm-linux-gnueabihf/libnss_dns.so.2 /tmp/
    
    Copy these off the device for local adjustment:
    scp IP_ADDR:/tmp/FILE .
    
  2. Decompress the initrd.img:
    cp initrd.img-3.16.0-4-armmp-lpae initrd.img-3.16.0-4-armmp-lpae.gzip
    gunzip initrd.img-3.16.0-4-armmp-lpae.gzip
    
  3. Make a new empty directory
    mkdir initramfs
    cd initramfs
    
  4. Unpack:
    sudo cpio -id < initrd.img-3.16.0-4-armmp-lpae
    
  5. Remove the old modules (LAVA can add these later, allowing tests to use an updated build with updated modules):
    sudo rm -rf ./lib/modules/*
    
  6. Start to customise - need a script for udhcpc and two of the libraries from the installed system to allow the initramfs to do DNS lookups successfully.
    cp ../libresolv.so.2 ./lib/arm-linux-gnueabihf/
    cp ../libnss_dns.so.2 ./lib/arm-linux-gnueabihf/
    
  7. Copy the udhcpc default script into place:
    mkdir ./etc/udhcpc/
    sudo cp ../udhcpc.d ./etc/udhcpc/default.script
    sudo chmod 0755 ./etc/udhcpc/default.script
    
  8. Rebuild the cpio archive:
    find .   cpio -H newc -o > ../initrd.img-armmp.cpio
    
  9. Recompress:
    cd ..
    gzip initrd.img-armmp.cpio
    
  10. If using u-boot, add the UBoot header:
    mkimage -A arm -T ramdisk -C none -d initrd.img-armmp.cpio.gz initrd.img-armmp.cpio.gz.u-boot
    
  11. Checksum the final file so that you can check that against the LAVA logs.
    md5sum initrd.img-armmp.cpio.gz.u-boot
    
Each type of device will need a set of modules modprobed before tests can start. With the refactoring code, I can use an inline YAML and use dmesg -n 5 to reduce the kernel message noise. The actual module names here are just those for the Cubietruck but by having these only in the job submission, it makes it easier to test particular combinations and requirements.
- dmesg -n 5
- lava-test-case udevadm --shell udevadm hwdb --update
- lava-test-case depmod --shell depmod -a
- lava-test-case sata-mod --shell modprobe -a stmmac ahci_sunxi sd_mod sg ext4
- lava-test-case ifconfig --shell ifconfig eth0 up
- lava-test-case udhcpc --shell udhcpc
- dmesg -n 7
In due course, this will be added to the main LAVA documentation to allow others to keep the initramfs up to date and to support further test development.

13 April 2015

Santiago Garc a Manti n: haproxy as a very very overloaded sslh

After using haproxy at work for some time I realized that it can be configured for a lot of things, for example: it knows about SNI (on ssl is the method we use to know what host the client is trying to reach so that we know what certificate to present and thus we can multiplex several virtual hosts on the same ssl IP:port) and it also knows how to make transparent proxy connections (the connections go through haproxy but the ending server will think they are arriving directly from the client, as it will see the client's IP as the source IP of the packages).With this two little features, which are available on haproxy 1.5 (Jessie's version has them all), I thought I could give it a try to substitute sslh with haproxy giving me a lot of possibilities that sslh cannot do.Having this in mind I thought I could multiplex several ssl services, not only https but also openvpn or similar, on the 443 port and also allow this services to arrive transparently to the final server. Thus what I wanted was not to mimic sslh (which can be done with haproxy) but to get the semantic I needed, which is similar to sslh but with more power and with a little different behaviour, cause I liked it that way.There is however one caveat that I don't like about this setup and it is that to achieve the transparency one has to run haproxy as root, which is not really something one likes :-( so, having transparency is great, but we'll be taking some risks here which I personally don't like, to me it isn't worth it.Anyway, here is the setup, it basically consists of a setup on haproxy but if we want transparency we'll have to add to it a routing and iptables setup, I'll describe here the whole setupHere is what you need to define on /etc/haproxy/haproxy.cfg:frontend ft_ssl bind 192.168.0.1:443 mode tcp option tcplog tcp-request inspect-delay 5s tcp-request content accept if req_ssl_hello_type 1 acl sslvpn req_ssl_sni -i vpn.example.net use_backend bk_sslvpn if sslvpn use_backend bk_web if req_ssl_sni -m found default_backend bk_ssh backend bk_sslvpn mode tcp source 0.0.0.0 usesrc clientip server srvvpn vpnserver:1194 backend bk_web mode tcp source 0.0.0.0 usesrc clientip server srvhttps webserver:443 backend bk_ssh mode tcp source 0.0.0.0 usesrc clientip server srvssh sshserver:22 An example of a transparent setup can be found here but lacks some details, for example, if you need to redirect the traffic to the local haproxy you'll want to use the xt_TPROXY, there is a better doc for that at squid's wiki. Anyway, if you are playing just with your own machine, like we typically do with sslh, you won't need the TPROXY power, as packets will come straight to your 443, so haproxy will be able to get the without any problem. The problem will come if you are using transparency (source 0.0.0.0 usesrc clientip) because then packets coming out of haproxy will be carrying the ip of the real client, and thus the answers of the backend will go to that client (but with different ports and other tcp data), so it will not work. We'll have to get those packets back to haproxy, for that what we'll do is mark the packages with iptables and then route them to the loopback interface using advanced routing. This is where all the examples will tell you to use iptables' mangle table with rules marking on PREROUTING but that won't work out if you are having all the setup (frontend and backends) in just one box, instead you'll have to write those rules to work on the OUTPUT chain of the mangle table, having something like this:*mangle :PREROUTING ACCEPT :INPUT ACCEPT :FORWARD ACCEPT :OUTPUT ACCEPT :POSTROUTING ACCEPT :DIVERT - -A OUTPUT -s public_ip -p tcp --sport 22 -o public_iface -j DIVERT -A OUTPUT -s public_ip -p tcp --sport 443 -o public_iface -j DIVERT -A OUTPUT -s public_ip -p tcp --sport 1194 -o public_iface -j DIVERT -A DIVERT -j MARK --set-mark 1 -A DIVERT -j ACCEPT COMMIT Take that just as an example, better suggestions on how to know what traffic to send to DIVERT are welcome. The point here is that if you are sending the service to some other box you can do it on PREROUTIING, but if you are sending the service to the very same box of haproxy you'll have to mark the packages on the OUTPUT chain.Once we have the packets marked we just need to route them, something like this will work out perfectly:ip rule add fwmark 1 lookup 100 ip route add local 0.0.0.0/0 dev lo table 100 And that's all for this crazy setup. Of course, if, like me, you don't like the root implication of the transparent setup, you can remove the "source 0.0.0.0 usesrc clientip" lines on the backends and forget about transparency (connections to the backend will come from your local IP), but you'll be able to run haproxy with dropped privileges and you'll just need the plain haproxy.cfg setup and not the weird iptables and advanced routing setup.Hope you like the article, btw, I'd like to point out the main difference of this setup vs sslh, it is that I'm only sending the packages to the ssl providers if the client is sending SNI info, otherwise I'm sending them to the ssh server, while sslh will send ssl clients without SNI also to the ssl provider. If your setup mimics sslh and you want to comment on it, feel free to do it.

10 April 2015

Richard Hartmann: Release Critical Bug report for Week 15

Still on the road with shittynet; sorry for missing last week. The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze and Wheezy release cycles?
Week Squeeze Wheezy Jessie
43 284 (213+71) 468 (332+136) 319 (240+79)
44 261 (201+60) 408 (265+143) 274 (224+50)
45 261 (205+56) 425 (291+134) 295 (229+66)
46 271 (200+71) 401 (258+143) 427 (313+114)
47 283 (209+74) 366 (221+145) 342 (260+82)
48 256 (177+79) 378 (230+148) 274 (189+85)
49 256 (180+76) 360 (216+155) 226 (147+79)
50 204 (148+56) 339 (195+144) ???
51 178 (124+54) 323 (190+133) 189 (134+55)
52 115 (78+37) 289 (190+99) 147 (112+35)
1 93 (60+33) 287 (171+116) 140 (104+36)
2 82 (46+36) 271 (162+109) 157 (124+33)
3 25 (15+10) 249 (165+84) 172 (128+44)
4 14 (8+6) 244 (176+68) 187 (132+55)
5 2 (0+2) 224 (132+92) 175 (124+51)
6 release! 212 (129+83) 161 (109+52)
7 release+1 194 (128+66) 147 (106+41)
8 release+2 206 (144+62) 147 (96+51)
9 release+3 174 (105+69) 152 (101+51)
10 release+4 120 (72+48) 112 (82+30)
11 release+5 115 (74+41) 97 (68+29)
12 release+6 93 (47+46)
13 release+7 50 (24+26)
14 release+8 51 (32+19)
15 release+9 39 (32+7)
16 release+10 20 (12+8)
17 release+11 24 (19+5)
18 release+12 2 (2+0)
Graphical overview of bug stats thanks to azhag:

9 April 2015

Mario Lang: A C++ sample collection

I am one of those people that best learns from looking at examples. No matter if I am trying to learn a programming pattern/idiom, or a completely new library or framework. Documentation is good (if it is good!) for diving into the details, but to get me started, I always want to look at a self contained example so that I can get a picture of the thing in my head. So I was very excited when a few days ago, CppSamples was announced on the ISO C++ Blog. While it is a very young site, it already contains some very useful gems. It is maintained over at GitHub, so it is also rather easy to suggest new additions, or improve the existing examples by submitting a pull request. Give it a try, it is really quite nice. In my book, the best resource I have found so far in 2015. BTW, Debian has a standard location for finding examples provided by a package. It is /usr/share/doc/<package>/examples/. I consider that very useful.

6 April 2015

Daniel Leidert: Motion picture capturing: Debian + motion + Logitech C910

Woodpecker near the windowWinter time is a good time for some nature observation. Yesterday I had a woodpecker (picture) in front of my kitchen window. During the recent weeks there were long-tailed tits, a wren and other rarely seen birds. So I thought, it might be a good idea to capture some of these events :) I still own a Logitech C910 USB camera which allows HD video capturing up to 1080p. So I checked the web for some software that would begin video capturing in the case of motion detection and found motion, already available for Debian users. So I gave it a try. I tested all available resolutions of the camera together with the capturing results. I found that the resulting framerate of both the live stream and the captured video is highly depending on the resolution and some few configuration options. Below is a summary of my tests and the results I've achieved so far.

Logitech C910 HD cameraJust a bit of data regarding the camera. AFAIK it allows for fluent video streams up to 720p.
$ dmesg
[..]
usb 7-3: new high-speed USB device number 5 using ehci-pci
usb 7-3: New USB device found, idVendor=046d, idProduct=0821
usb 7-3: New USB device strings: Mfr=0, Product=0, SerialNumber=1
usb 7-3: SerialNumber: 91CF80A0
usb 7-3: current rate 0 is different from the runtime rate 16000
usb 7-3: current rate 0 is different from the runtime rate 32000
uvcvideo: Found UVC 1.00 device (046d:0821)
input: UVC Camera (046d:0821) as /devices/pci0000:00/0000:00:1a.7/usb7/7-3/7-3:1.2/input/input17

$ lsusb
[..]
Bus 007 Device 005: ID 046d:0821 Logitech, Inc. HD Webcam C910
[..]

$ v4l2-ctl -V -d /dev/video1
Format Video Capture:
Width/Height : 1280/720
Pixel Format : 'YUYV'
Field : None
Bytes per Line: 2560
Size Image : 1843200
Colorspace : SRGB
Also the uvcvideo kernel module is loaded and the user in question is part of the video group. Installation and startInstallation of the software is as easy as always:
apt-get install motion
It is possible to run the software as a service. But for testing, I copied /etc/motion/motion.conf to ~/.motion/motion.conf, fixed its permissions (you cannot copy the file as user - it's not world readable) and disabled the daemon mode.
daemon off
Note that in my case the correct device is /dev/video1 because the laptop has a built-in camera, that is /dev/video0. Also the target directory should be writeable by my user:
videodevice /dev/video1
target_dir ~/Videos
Then running motion from the command line ...
$ motion
[..]
[0] [NTC] [ALL] motion_startup: Motion 3.2.12+git20140228 Started
[..]
[1] [NTC] [ALL] motion_init: Thread 1 started , motion detection Enabled
[0] [NTC] [ALL] main: Thread 1 is device: /dev/video1 input -1
[1] [NTC] [VID] v4l2_get_capability:
------------------------
cap.driver: "uvcvideo"
cap.card: "UVC Camera (046d:0821)"
cap.bus_info: "usb-0000:00:1a.7-1"
cap.capabilities=0x84000001
------------------------
[1] [NTC] [VID] v4l2_get_capability: - VIDEO_CAPTURE
[1] [NTC] [VID] v4l2_get_capability: - STREAMING
[1] [NTC] [VID] v4l2_select_input: name = "Camera 1", type 0x00000002, status 00000000
[1] [NTC] [VID] v4l2_select_input: - CAMERA
[..]
[1] [NTC] [ALL] image_ring_resize: Resizing pre_capture buffer to 1 items
... will begin to capture motion detection events and also output a live stream. CTRL+C will stop it again. Live streamThe live stream is available by pointing the browser to localhost:8081. However, the stream seems to run at 1 fps (frames per second) and indeed does. The stream gets more quality by this configuration:
stream_motion on
stream_maxrate 100
The first option is responsible, that the stream only runs at one fps if there is no motion detection event. Otherwise the framerate increases to its maximum value, which is either the one given by stream_maxrate or the camera limit. The quality of the stream picture can be increased a bit further too by increasing the stream_quality value. Because I neither need the stream nor the control feed I disabled both:
stream_port 0
webcontrol_port 0
Picture capturingBy default there is video and picture capturing if a motion event is detected. I'm not interested in these pictures, so I turned them off:
output_pictures off
FYI: If you want a good picture quality, then the value of quality should very probably be increased. Video capturingThis is the really interesting part :) Of course if I will "shoot" some birds (with the camera), then a small image of say 320x240 pixels is not enough. The camera allows for a capture resolution up to 1920x1080 pixels (1080p). It is advertised for fluent video streams up to 720p (1280x720 pixels). So I tried the following resolutions: 320x240, 640x480, 800x600, 640x360 (360p), 1280x720 (720p) and 1920x1080 (1080p). These are easily configured by the width and height variables. For example the following configures motion for 1280x720 pixels (720p):
width 1280
height 720
The result was really disappointing. No event is captured with more then 20 fps. At higher resolutions the framerate drops even further and at the highest resolution of 1920x1080 pixels, the framerate is only two(!) fps. Also every created video runs much too fast and even faster by increasing the framerate variable. Of course its default value of 2 (fps) is not enough for fluent videos. AFAIK the C910 can run with 30 fps at 1280x720 pixels. So increasing the value of framerate, the maximum framerate recorded, is a must-do. (If you wanna test yourself, check the log output for the value following event_new_video FPS.)The solution to the issue, that videos are running too fast, however is to increase the pre_capture value, the number of pre-captured (buffered) pictures from before motion was detected. Even small values like 3..5 result in a distinctive improvement of the situation. Though increasing the value further didn't have any effect. So the values below should almost get the most out of the camera and result in videos in normal speed.
framerate 100
pre_capture 5
Videos in 1280x720 pixels are still captured with 10 fps and I don't know why. Running guvcview, the same camera allows for 30 fps in this resolution (even 60 fps in lower resolutions). However, even if the framerate could be higher, the resulting video runs fluently. Still the quality is just moderate (or to be honest, still disappointing). It looks "pixelated". Only static pictures are sharp. It took me a while to fix this too, because I first thought, the reason is the camera or missing hardware support. It is not :) The reason is, that ffmpeg is configured to produce a moderate(?)-quality video. The relevant variables are ffmpeg_bps and ffmpeg_variable_bitrate. I got the best results just changing the latter:
ffmpeg_variable_bitrate 2
Finally the resulting video quality is promising. I'll start with this configuration setting up an observation camera for the bird feeding ground.There is one more tweak for me. I got even better results by enabling the auto_brightness feature.
auto_brightness on
Complete configurationSo the complete configuration looks like this (only those options changed to the original config file)
daemon off
videodevice /dev/video1
width 1280
height 720
framerate 100
auto_brightness on
ffmpeg_variable_bitrate 2
target_dir /home/user/Videos
stream_port 0 #8081
stream_motion on
stream_maxrate 100
webcontrol_port 0 #8080
Links Continue with part II ...

1 April 2015

Benjamin Mako Hill: RomancR: The Future of the Sharing-Your-Bed Economy

romancer_logo Today, Aaron Shaw and I are pleased to announce a new startup. The startup is based around an app we are building called RomancR that will bring the sharing economy directly into your bedrooms and romantic lives. When launched, RomancR will bring the kind of market-driven convenience and efficiency that Uber has brought to ride sharing, and that AirBnB has brought to room sharing, directly into the most frustrating and inefficient domain of our personal lives. RomancR is Uber for romance and sex. Here s how it will work: Of course, there are many existing applications like Tinder and Grindr that help facilitate romance, dating, and hookups. Unfortunately, each of these still relies on old-fashion intrinsic ways of motivating people to participate in romantic endeavors. The sharing economy has shown us that systems that rely on these non-monetary motivations are ineffective and limiting! For example, many altruistic and socially-driven ride-sharing systems existed on platforms like Craigslist or Ridejoy before Uber. Similarly, volunteer-based communities like Couchsurfing and Hospitality Club existed for many years before AirBnB. None of those older systems took off in the way that their sharing economy counterparts were able to! The reason that Uber and AirBnB exploded where previous efforts stalled is that this new generation of sharing economy startups brings the power of markets to bear on the problems they are trying to solve. Money both encourages more people to participate in providing a service and also makes it socially easier for people to take that service up without feeling like they are socially in debt to the person providing the service for free. The result has been more reliable and effective systems for proving rides and rooms! The reason that the sharing economy works, fundamentally, is that it has nothing to do with sharing at all! Systems that rely on people s social desire to share without money projects like Couchsurfing are relics of the previous century. RomancR, which we plan to launch later this year, will bring the power and efficiency of markets to our romantic lives. You will leave your pitiful dating life where it belongs in the dustbin of history! Go beyond antiquated non-market systems for finding lovers. Why should we rely on people s fickle sense of taste and attractiveness, their complicated ideas of interpersonal compatibility, or their sense of altruism, when we can rely on the power of prices? With RomancR, we won t have to! Note: Thanks to Yochai Benkler whose example of how leaving a $100 bill on the bedside table of a person with whom you spent the night can change the nature of the a romantic interaction inspired the idea for this startup.

Benjamin Mako Hill: RomancR: The Future of the Sharing-Your-Bed Economy

romancer_logo Today, Aaron Shaw and I are pleased to announce a new startup. The startup is based around an app we are building called RomancR that will bring the sharing economy directly into your bedrooms and romantic lives. When launched, RomancR will bring the kind of market-driven convenience and efficiency that Uber has brought to ride sharing, and that AirBnB has brought to room sharing, directly into the most frustrating and inefficient domain of our personal lives. RomancR is Uber for romance and sex. Here s how it will work: Of course, there are many existing applications like Tinder and Grindr that help facilitate romance, dating, and hookups. Unfortunately, each of these still relies on old-fashion intrinsic ways of motivating people to participate in romantic endeavors. The sharing economy has shown us that systems that rely on these non-monetary motivations are ineffective and limiting! For example, many altruistic and socially-driven ride-sharing systems existed on platforms like Craigslist or Ridejoy before Uber. Similarly, volunteer-based communities like Couchsurfing and Hospitality Club existed for many years before AirBnB. None of those older systems took off in the way that their sharing economy counterparts were able to! The reason that Uber and AirBnB exploded where previous efforts stalled is that this new generation of sharing economy startups brings the power of markets to bear on the problems they are trying to solve. Money both encourages more people to participate in providing a service and also makes it socially easier for people to take that service up without feeling like they are socially in debt to the person providing the service for free. The result has been more reliable and effective systems for proving rides and rooms! The reason that the sharing economy works, fundamentally, is that it has nothing to do with sharing at all! Systems that rely on people s social desire to share without money projects like Couchsurfing are relics of the previous century. RomancR, which we plan to launch later this year, will bring the power and efficiency of markets to our romantic lives. You will leave your pitiful dating life where it belongs in the dustbin of history! Go beyond antiquated non-market systems for finding lovers. Why should we rely on people s fickle sense of taste and attractiveness, their complicated ideas of interpersonal compatibility, or their sense of altruism, when we can rely on the power of prices? With RomancR, we won t have to! Note: Thanks to Yochai Benkler whose example of how leaving a $100 bill on the bedside table of a person with whom you spent the night can change the nature of the a romantic interaction inspired the idea for this startup.

30 March 2015

Matthias Klumpp: Limba Project: Another progress report

And once again, it s time for another Limba blogpost :-)limba-small Limba is a solution to install 3rd-party software on Linux, without interfering with the distribution s native package manager. It can be useful to try out different software versions, use newer software on a stable OS release or simply to obtain software which does not yet exist for your distribution. Limba works distribution-independent, so software authors only need to publish their software once for all Linux distributions. I recently released version 0.4, with which all most important features you would expect from a software manager are complete. This includes installing & removing packages, GPG-signing of packages, package repositories, package updates etc. Using Limba is still a bit rough, but most things work pretty well already. So, it s time for another progress report. Since a FAQ-like list is easier to digest. compared to a long blogpost, I go with this again. So, let s address one important general question first: How does Limba relate to the GNOME Sandboxing approach? (If you don t know about GNOMEs sandboxes, take a look at the GNOME Wiki Alexander Larsson also blogged about it recently) First of all: There is no rivalry here and no NIH syndrome involved. Limba and GNOMEs Sandboxes (XdgApp) are different concepts, which both have their place. The main difference between both projects is the handling of runtimes. A runtime is the shared libraries and other shared ressources applications use. This includes libraries like GTK+/Qt5/SDL/libpulse etc. XdgApp applications have one big runtime they can use, built with OSTree. This runtime is static and will not change, it will only receive critical security updates. A runtime in XdgApp is provided by a vendor like GNOME as a compilation of multiple single libraries. Limba, on the other hand, generates runtimes on the target system on-the-fly out of several subcomponents with dependency-relations between them. Each component can be updated independently, as long as the dependencies are satisfied. The individual components are intended to be provided by the respective upstream projects. Both projects have their individual up and downsides: While the static runtime of XdgApp projects makes testing simple, it is also harder to extend and more difficult to update. If something you need is not provided by the mega-runtime, you will have to provide it by yourself (e.g. we will have some applications ship smaller shared libraries with their binaries, as they are not part of the big runtime). Limba does not have this issue, but instead, with its dynamic runtimes, relies on upstreams behaving nice and not breaking ABIs in security updates, so existing applications continue to be working even with newer software components. Obviously, I like the Limba approach more, since it is incredibly flexible, and even allows to mimic the behaviour of GNOMEs XdgApp by using absolute dependencies on components. Do you have an example of a Limba-distributed application? Yes! I recently created a set of package for Neverball Alexander Larsson also created a XdgApp bundle for it, and due to the low amount of stuff Neverball depends on, it was a perfect test subject. One of the main things I want to achieve with Limba is to integrate it well with continuous integration systems, so you can automatically get a Limba package built for your application and have it tested with the current set of dependencies. Also, building packages should be very easy, and as failsafe as possible. You can find the current Neverball test in the Limba-Neverball repository on Github. All you need (after installing Limba and the build dependencies of all components) is to run the make_all.sh script. Later, I also want to provide helper tools to automatically build the software in a chroot environment, and to allow building against the exact version depended on in the Limba package. Creating a Limba package is trivial, it boils down to creating a simple control file describing the dependencies of the package, and to write an AppStream metadata file. If you feel adventurous, you can also add automatic build instructions as a YAML file (which uses a subset of the Travis build config schema) This is the Neverball Limba package, built on Tanglu 3, run on Fedora 21: Limba-installed Neverball Which kernel do I need to run Limba? The Limba build tools run on any Linux version, but to run applications installed with Limba, you need at least Linux 3.18 (for Limba 0.4.2). I plan to bump the minimum version requirement to Linux 4.0+ very soon, since this release contains some improvements in OverlayFS and a few other kernel features I am thinking about making use of. Linux 3.18 is included in most Linux distributions released in 2015 (and of course any rolling release distribution and Fedora have it). Building all these little Limba packages and keeping them up-to-date is annoying Yes indeed. I expect that we will see some bigger Limba packages bundling a few dependencies, but in general this is a pretty annoying property of Limba currently, since there are so few packages available you can reuse. But I plan to address this. Behind the scenes, I am working on a webservice, which will allow developers to upload Limba packages. This central ressource can then be used by other developers to obtain dependencies. We can also perform some QA on the received packages, map the available software with CVE databases to see if a component is vulnerable and publish that information, etc. All of this is currently planned, and I can t say a lot more yet. Stay tuned! (As always: If you want to help, please contact me) Are the Limba interfaces stable? Can I use it already? The Limba package format should be stable by now since Limba is still Alpha software, I will however, make breaking changes in case there is a huge flaw which makes it reasonable to break the IPK package format. I don t think that this will happen though, as the Limba packages are designed to be easily backward- and forward compatible. For the Limba repository format, I might make some more changes though (less invasive, but you might need to rebuilt the repository). tl;dr: Yes! Plase use Limba and report bugs, but keep in mind that Limba is still in an early stage of development, and we need bug reports! Will there be integration into GNOME-Software and Muon? From the GNOME-Software side, there were positive signals about that, but some technical obstancles need to be resolved first. I did not yet get in contact with the Muon crew they are just implementing AppStream, which is a prerequisite for having any support for Limba[1]. Since PackageKit dropped the support for plugins, every software manager needs to implement support for Limba.
So, thanks for reading this (again too long) blogpost :) There are some more exciting things coming soon, especially regarding AppStream on Debian/Ubuntu! [1]: And I should actually help with the AppStream support, but currently I can not allocate enough time to take that additional project as well this might change in a few weeks. Also, Muon does pretty well already!

Dimitri John Ledkov: Boiling frog, or when did we loose it with /etc ?

$ sudo find /etc -type f wc -l
2794

StatelessWhen was the last time you looked at /etc and thought - "I honestly know what every single file in here is". Or for example had a thought "Each file in here is configuration changes that I made". Or for example do you have confidence that your system will continue to function correctly if any of those files and directories are removed?

Traditionally most *NIX utilities are simple enough utilities, that do not require any configuration files what's so ever. However most have command line arguments, and environment variables to manipulate their behavior. Some of the more complex utilities have configuration files under /etc, sometimes with "layer" configuration from user's home directory (~/). Most of them are generally widely accepted. However, these do not segregate upstream / distribution / site administrator / local administrator / user configuration changes. Most update mechanisms created various ways to deal with merging and maintaining the correct state of those. For example both dpkg & RPM (%config) have elaborate strategies and policies and ways to deal with them. However, even today, still, they cause problems: prompting user for whitespace changes in config files, not preserving user changes, or failing to migrate them.

I can't find exact date, but it has now been something like 12 years since XDG Base directory specification was drafted. It came from Desktop Environment requirements, but one thing it achieves is segregation between upstream / distro / admin / user induced changes. When applications started to implement Base directory specification, I started to feel empowered. Upstream ships sensible configs in /usr, distribution integrators ship their overlay tweaks packaged in /usr, my site admin applies further requirements in /etc, and as I user I am free to improve or brake everything with configs in ~/. One of the best things from this setup - no upgrade prompts, and ease of reverting each layer of those configs (or at least auditing where the settings are coming from).

However, the uptake of XDG Base directory spec is slow / non-existing among the core components of any OS today. And at the same time /etc has grown to be a dumping ground for pretty much everything under the sun:
  • Symlink farms - E.g. /etc/rc*.d/*, /etc/systemd/system/*.wants/*, /etc/ssl/certs/*
  • Cache files - E.g. /etc/ld.so.cache
  • Empty (and mandatory) directories
  • Empty (and mandatory) "configuration" files. - E.g. whitespace & comments only
Let's be brutally honest and say that none of the above belongs in /etc. /etc must be for end-user configuration only, made by the end user alone and nobody else (or e.g. an automation tool driven by the end-user, like puppet).

Documentation of available configuration options and syntax to specify those in the config files should be shipped... in the documentation. E.g. man pages, /usr/share/doc, and so on. And not as the system-wide "example" config files. Absence of the files in /etc must not be treated as fatal, but a norm, since most users use default settings (especially for the most obscure options). Lastly compiled-in defaults should be used where possible, or e.g. layer configuration from multiple locations (e.g. /usr, /etc, ~/ where appropriate).

Above observations are not novel, and shared by most developers and users in the wider open source ecosystem. There are many projects and concepts to deal with this problem by using automation (e.g. puppet, chef), by migrating to new layouts (e.g. implementing / supporting XDG base dir spec), using "app bundles" (e.g. mobile apps, docker), or fully enumerating/abstracting everything in a generic manner (e.g. NixOS). Whilst fixing the issue at hand, these solutions do increase the dependency on files in /etc to be available. In other words we grew a de-facto user-space API we must not break, because modifications to the well known files in /etc are expected to take effect by both users and many administrator tools.

Since August last year, I have joined Open Source Technology Center at Intel, and have been working on Clear Linux* Project for Intel Architecture. One of the goals we have set out is to achieve stateless operation - that is to have empty /etc by default, reserved for user modification alone, yet continuing to support all legacy / well-known configuration paths. The premise is that all software can be patched with auto-detection, built-in defaults or support for layered configuration to achieve this. I hope that this work would interest everyone and will be widely adopted.

Whilst the effort to convert everything is still on going, I want to discuss a few examples of any core system.

ShadowThe login(1) command, whilst having built-in default for every single option exits with status 1, if it cannot stat(2) login.defs(5) file.

The passwd(1) command will write out the salted/hashed password in the passwd(5) file, rather than in shadow(5), if it cannot stat the shadow(5) file. There is similar behavior with gshadow. I found it very ironic, that upstream project "shadow" does not use shadow(5) by default.

Similarly, stock files manipulated by passwd/useradd/groupadd utilities are not created, if missing.

Some settings in login.defs(5) are not applicable, when compiled with PAM support, yet present in the default shipped login.defs(5) file.

Patches to resolve above issues are undergoing review on the upstream mailing list.
DBusIn xml based configuration, includedir' elements are mandatory to exist on disk, that is empty directory must be present, if referenced. If these directories are non-existant, the configuration fails to load and the system or session bus are not started.

Similarly, upstream have general agreement with the stateless concept and patches to move all of dbus default configurations from /etc to /usr are being reviewed for inclusion at the bug tracker. I hope this change will make into the 1.10 stable release.

GNU Lib CToday, we live in a dual-stack IPv4 and IPv6 world, where even the localhost has multiple IP addresses. As a slightly ageist time reference, the first VCS I ever used was git. Thus when I read below, I get very confused:
$ cat /etc/host.conf
# The "order" line is only used by old versions of the C library.
order hosts,bind
multi on
Why not simply do this:
--- a/resolv/res_hconf.c
+++ b/resolv/res_hconf.c
@@ -309,6 +309,8 @@ do_init (void)
if (hconf_name == NULL)
hconf_name = _PATH_HOSTCONF;

+ arg_bool (ENV_MULTI, 1, "on", HCONF_FLAG_MULTI);
+
fp = fopen (hconf_name, "rce");
if (fp)

There are still many other packages that needed fixes similar to above. Stay tuned for further stateless observations about Glibc, OpenSSH, systemd and other well known packages.

In the mean time, you can try out https://clearlinux.org/ images that implement above and more already. If you want to chat about it more, comment on G+, find myself on irc - xnox @ irc.freenode.net #clearlinux and join our mailing list to kick the conversation off, if you are interested in making the world more stateless.

ps.
I am a professional Linux Distribution developer, currently employed by Intel, however the postings on this site are my own and don't necessarily represent Intel's or any other past/present/future employer positions, strategies, or opinions.

* Other names and brands may be claimed as the property of others


28 March 2015

Matt Zimmerman: What I think about thought

Only parts of us will ever
touch o n l y parts of others
one s own truth is just that really one s own truth.
We can only share the part that is u n d e r s t o o d b y within another s knowing acceptable t o t h e o t h e r t h e r e f o r e so one
is for most part alone.
As it is meant to be in
evidently in nature at best t h o u g h perhaps it could make
our understanding seek
another s loneliness out.
unpublished poem by Marilyn Monroe, via berlin-artparasites This poem inspired me to put some ideas into words this morning, an attempt to summarize my current working theory of consciousness. Ideas travel through space and time. An idea that exists in my mind is filtered through my ability to express it somehow (words, art, body language, ), and is then interpreted by your mind and its models for understanding the world. This shifts your perspective in some way, some or all of which may be unconscious. When our minds encounter new ideas, they are accepted or rejected, reframed, and integrated with our existing mental models. This process forms a sort of living ecosystem, which maintains equilibrium within the realm of thought. Ideas are born, divide, mutate, and die in the process. Language, culture, education and so on are stable structures which form and support this ecosystem. Consciousness also has analogues of the immune system, for example strongly held beliefs and models which tend to reject certain ideas. Here again these can be unconscious or conscious. I ve seen it happen that if someone hears an idea they simply cannot integrate, they will behave as if they did not hear it at all. Some ideas can be identified as such a serious threat that ignoring them is not enough to feel safe: we feel compelled to eliminate the idea in the external world. The story of Christianity describes a scenario where an idea was so threatening to some people that they felt compelled to kill someone who expressed it. A microcosm of this ecosystem also exists within each individual mind. There are mental structures which we can directly introspect and understand, and others which we can only infer by observing our thoughts and behaviors. These structures communicate with each other, and this communication is limited by their ability to speak each other s language . A dream, for example, is the conveyance of an idea from an unconscious place to a conscious one. Sometimes we get the message, and sometimes we don t. We can learn to interpret, but we can t directly examine and confirm if we re right. As in biology, each part of this process introduces uncountable errors , but the overall system is surprisingly robust and stable. This whole system, with all its many minds interacting, can be thought of as an intelligence unto itself, a gestalt consciousness. This interpretation leads to some interesting further conclusions: Naturally, this is by no means an original idea (can such a thing exist?). It is my own take on the subject, informed both consciously and unconsciously by my own study, first-hand experience, conversations I ve had with others, and so on. It s informed by the countless thinkers who have influenced me. Its expression is limited by my ability to write about it in a way that makes sense to other people.
Maybe some of this makes sense to you, and maybe I seem insane, or maybe both. Hopefully you don t find that you have an inexplicable unconscious desire to kill me!

27 March 2015

Richard Hartmann: Release Critical Bug report for Week 13

The UDD bugs interface currently knows about the following release critical bugs: How do we compare to the Squeeze and Wheezy release cycles?
Week Squeeze Wheezy Jessie
43 284 (213+71) 468 (332+136) 319 (240+79)
44 261 (201+60) 408 (265+143) 274 (224+50)
45 261 (205+56) 425 (291+134) 295 (229+66)
46 271 (200+71) 401 (258+143) 427 (313+114)
47 283 (209+74) 366 (221+145) 342 (260+82)
48 256 (177+79) 378 (230+148) 274 (189+85)
49 256 (180+76) 360 (216+155) 226 (147+79)
50 204 (148+56) 339 (195+144) ???
51 178 (124+54) 323 (190+133) 189 (134+55)
52 115 (78+37) 289 (190+99) 147 (112+35)
1 93 (60+33) 287 (171+116) 140 (104+36)
2 82 (46+36) 271 (162+109) 157 (124+33)
3 25 (15+10) 249 (165+84) 172 (128+44)
4 14 (8+6) 244 (176+68) 187 (132+55)
5 2 (0+2) 224 (132+92) 175 (124+51)
6 release! 212 (129+83) 161 (109+52)
7 release+1 194 (128+66) 147 (106+41)
8 release+2 206 (144+62) 147 (96+51)
9 release+3 174 (105+69) 152 (101+51)
10 release+4 120 (72+48) 112 (82+30)
11 release+5 115 (74+41) 97 (68+29)
12 release+6 93 (47+46) 87 (71+16)
13 release+7 50 (24+26) 97 (77+20)
14 release+8 51 (32+19)
15 release+9 39 (32+7)
16 release+10 20 (12+8)
17 release+11 24 (19+5)
18 release+12 2 (2+0)
Graphical overview of bug stats thanks to azhag:

26 March 2015

Zlatan Todori : Random bits

Gogs I installed today Gogs and configured it with mysql (yes, yes, I know - use postgres you punk!). I will not post details of how I did it because:
  • It still has "weird" coding as pointed already by others
  • It doesn't have fork and pull request ability yet
And there was end of journey. When they code in fork/PR , I will close my eyes on other coding stuff and try it again because Gitlab is not close to my heart and installing their binary takes ~850MB of space which means a lot of ruby code that could go wrong way. It would be really awesome to have in archive something to apt install and have github-like place. It would be great if Debian infrastructure would have the possibility to have that.

Diaspora* Although I am thrilled about it finally reaching Debian archive, it still isn't ready. Not even closely. I couldn't even finish installation of it and it's not suitable for main archive as it takes files from github repo of diaspora. Maybe poking around Bitnami folks about how they did it.

The power of Free software Text Secure is was an mobile app that I thought it could take on Viber or WhatsUp. Besides all its goodies it had chance to send encrypted SMS to other TS users. Not anymore. Fortunate, there is a fork called SMSSecure which still has that ability.

Trolls So there is this Allwinner company that does crap after crap. Their latest will reach wider audience and I hope it gets resolved in a matter how they would react if some big proprietary company was stealing their code. It seems Allwinner is a pseudo for Alllooser. Whoa, that was fun!

A year old experiment So I had a bet with a friend that I will run for a year Debian Unstable mixed with some packages from experimental and do some random testings on packages of interest to them. Also I promised to update aggressively so it was to be twice a day. This was my only machine so the bet was really good as it by theory could break very often. Well on behalf of Debian community, I can say that Debian hasn't had a single big breakage. Yay! The good side: on average I had ~3000 packages installed (they were in range from 2500-3500). I had for example xmonad, e17, gnome, cinnamon, xfce, systemd from experimental, kernels from experimental, nginx, apache, a lot of heavy packages, mixed packages from pip, npm, gems etc. So that makes it even more incredible that it stayed stable. There is no bigger kudos to people working on Debian, then when some sadist tries countless of ways to break it and Debian is just keeps running. I mean, I was doing my $PAID_WORK on this machine! The bad side: there were small breakages. It's seems that polkit and systemd-side of gnome were going through a lot of changes because sometimes system would ask password for every action (logout, suspend, poweroff, connect to network etc), audio would work and would not work, would often by itself just mute sound on every play or it would take it to 100% (which would blow my head when I had earplugs), bluetooth is almost de facto not working in gnome (my bluetooth mice worked without single problem in lenny, squeeze, in wheezy it maybe had once or twice a problem, but in this year long test it's almost useless). System would also have random hangs from time to time. The test: in the beginning my radeon card was too new and it was not supported by FLOSS driver so I ended up using fglrx which caused me a lot of annoyance (no brightness control, flickering of screen) but once FLOSS driver got support I was on it, and it performed more fluid (no glitches while moving windows). So as my friends knew that I have radeon and they want to play games on their machines (I play my Steam games on FLOSS driver) they set me the task to try fglrx driver every now end then. End result - there is no stable fglrx driver for almost a year, it breaks graphical interface so I didn't even log into DE with it for at least 8 months if not more. On the good side my expeditions in flgrx where quick - install it, boot into disaster, remove it, boot into freedom. Downside seems to be that removing fglrx driver, leaves a lot of its own crap on system (I may be mistaking but it seems I am not). Debian with love Well, that's all for today. I think so. You can never be sure.

Zlatan Todori : Random bits

Gogs I installed today Gogs and configured it with mysql (yes, yes, I know - use postgres you punk!). I will not post details of how I did it because:
  • It still has "weird" coding as pointed already by others
  • It doesn't have fork and pull request ability yet
And there was end of journey. When they code in fork/PR , I will close my eyes on other coding stuff and try it again because Gitlab is not close to my heart and installing their binary takes ~850MB of space which means a lot of ruby code that could go wrong way. It would be really awesome to have in archive something to apt install and have github-like place. It would be great if Debian infrastructure would have the possibility to have that.

Diaspora* Although I am thrilled about it finally reaching Debian archive, it still isn't ready. Not even closely. I couldn't even finish installation of it and it's not suitable for main archive as it takes files from github repo of diaspora. Maybe poking around Bitnami folks about how they did it.

The power of Free software Text Secure is was an mobile app that I thought it could take on Viber or WhatsUp. Besides all its goodies it had chance to send encrypted SMS to other TS users. Not anymore. Fortunate, there is a fork called SMSSecure which still has that ability.

Trolls So there is this Allwinner company that does crap after crap. Their latest will reach wider audience and I hope it gets resolved in a matter how they would react if some big proprietary company was stealing their code. It seems Allwinner is a pseudo for Alllooser. Whoa, that was fun!

A year old experiment So I had a bet with a friend that I will run for a year Debian Unstable mixed with some packages from experimental and do some random testings on packages of interest to them. Also I promised to update aggressively so it was to be twice a day. This was my only machine so the bet was really good as it by theory could break very often. Well on behalf of Debian community, I can say that Debian hasn't had a single big breakage. Yay! The good side: on average I had ~3000 packages installed (they were in range from 2500-3500). I had for example xmonad, e17, gnome, cinnamon, xfce, systemd from experimental, kernels from experimental, nginx, apache, a lot of heavy packages, mixed packages from pip, npm, gems etc. So that makes it even more incredible that it stayed stable. There is no bigger kudos to people working on Debian, then when some sadist tries countless of ways to break it and Debian is just keeps running. I mean, I was doing my $PAID_WORK on this machine! The bad side: there were small breakages. It's seems that polkit and systemd-side of gnome were going through a lot of changes because sometimes system would ask password for every action (logout, suspend, poweroff, connect to network etc), audio would work and would not work, would often by itself just mute sound on every play or it would take it to 100% (which would blow my head when I had earplugs), bluetooth is almost de facto not working in gnome (my bluetooth mice worked without single problem in lenny, squeeze, in wheezy it maybe had once or twice a problem, but in this year long test it's almost useless). System would also have random hangs from time to time. The test: in the beginning my radeon card was too new and it was not supported by FLOSS driver so I ended up using fglrx which caused me a lot of annoyance (no brightness control, flickering of screen) but once FLOSS driver got support I was on it, and it performed more fluid (no glitches while moving windows). So as my friends knew that I have radeon and they want to play games on their machines (I play my Steam games on FLOSS driver) they set me the task to try fglrx driver every now end then. End result - there is no stable fglrx driver for almost a year, it breaks graphical interface so I didn't even log into DE with it for at least 8 months if not more. On the good side my expeditions in flgrx where quick - install it, boot into disaster, remove it, boot into freedom. Downside seems to be that removing fglrx driver, leaves a lot of its own crap on system (I may be mistaking but it seems I am not). Debian with love Well, that's all for today. I think so. You can never be sure.

24 March 2015

Daniel Pocock: The easiest way to run your own OpenID provider?

A few years ago, I was looking for a quick and easy way to run OpenID on a small web server. A range of solutions were available but some appeared to be slightly more demanding than what I would like. For example, one solution required a servlet container such as Tomcat and another one required some manual configuration of Python with Apache. I came across the SimpleID project. As the name implies, it is simple. It is written in PHP and works with the Apache/PHP environment on just about any Linux web server. It allows you to write your own plugin for a user/password database or just use flat files to get up and running quickly with no database at all. This seemed like the level of simplicity I was hoping for so I created the Debian package of SimpleID. SimpleID is also available in Ubuntu. Help needed Thanks to a contribution from Jean-Michel Nirgal Vourg re, I've just whipped up a 0.8.1-14 package that should fix Apache 2.4 support in jessie. I also cleaned up a documentation bug and the control file URLs. Nonetheless, it may be helpful to get feedback from other members of the community about the future of this package:
  • Is it considered secure enough?
  • Have other people found it relatively simple to install or was I just lucky when I tried it?
  • Are there other packages that now offer such a simple way to get OpenID for a vanilla Apache/PHP environment?
  • Would anybody else be interested in helping to maintain this package?
  • Would anybody like to see this packaged in other distributions such as Fedora?
  • Is anybody using it for any online community?
Works with HOTP one-time-passwords and LDAP servers One reason I chose SimpleID is because of dynalogin, the two-factor authentication framework. I wanted a quick and easy way to use OTP with OpenID so I created the SimpleID plugin for dynalogin, also available as a package. I also created the LDAP backend for SimpleID, that is available as a package too. Works with Drupal I tested SimpleID for login to a Drupal account when the OpenID support is enabled in Drupal, it worked seamlessly. I've also tested it with a few public web sites that support OpenID.

Russ Allbery: Review: Fukushima

Review: Fukushima, by David Lochbaum, et al.
Author: David Lochbaum
Author: Edwin Lyman
Author: Susan Q. Stranahan
Author: Union of Concerned Scientists
Publisher: The New Press
Copyright: 2014
ISBN: 1-59558-927-9
Format: Kindle
Pages: 320
This is a very interesting book, and I can recommend it, but there are two things you should be aware of up-front. The packaging does not necessarily make clear what expectations you should have of it going in. First, the subtitle (The Story of a Nuclear Disaster) should have appended to it And Its Implications for US Nuclear Power Policy. This book is very concerned with the impact of the Fukushima disaster on US policy and nuclear regulation, to the point where I think more than half of the book is about US agencies, nuclear regulatory history, and US reaction. There's nothing wrong with that, of course: the US should take a hard look at its own nuclear energy policy given the events at Fukushima, and it's a worthy topic for a book. But if you go into this book expecting a broader perspective, you will be disappointed. For example, I think the fact that France has a lot of nuclear power was mentioned maybe twice in the whole book, and French reaction was never discussed at all. There is a very detailed examination of exactly what happened at Fukushima (more on that in a moment), but most of the policy implications are examined purely from a US perspective. Even Japanese nuclear policy gets somewhat short shrift. Second, note that the fourth listed co-author is the Union of Concerned Scientists. For those not familiar with US environmental groups, the UCS has a reputation as an anti-nuclear advocacy organization. I don't think that's entirely fair; I think the UCS's position on nuclear power is better summarized as holding that it is theoretically possible to run a nuclear power plant safely, but the actual US nuclear power industry is not very close to that standard, and it would require much tighter regulation and more investment in safety systems to reach that standard. But be aware that the authors of this book have a clear position on the adequacy of current nuclear power safety standards, namely that they aren't. And they don't try to conceal that position in this book. Personally, I prefer authors to be open about their perspective in books like this, but your mileage may vary. There, disclaimers out of the way. I bought this book for a specific reason: I had followed some of the news coverage at the time of the earthquake and tsunami, and then (like many people, I suspect) lost track of the final outcome as the story fell out of the news and I started ignoring people who didn't understand how large the Pacific Ocean is. Now that we've had the benefit of several years of analysis and thoughtful reconstruction of events, I wanted to know what had actually happened. I'm happy to say that this book delivers quite well on that front. Roughly the first half of the book is a detailed blow-by-blow description of exactly what happened at Fukushima, at least as well as we've been able to reconstruct, told as an engrossing and dramatic narrative. There may be a little too much interleaving of reactions within the US government, which I suspect will particularly annoy non-US readers, but the level of factual detail is excellent, clear, and well-explained. What I wasn't expecting, but was pleasantly surprised by, is that it's also a great story. There's tension, conflict, heroism, hard choices, and moral quandries, and the authors do a great job conveying factual information while still giving the reader the sense of being in the middle of the unfolding drama. They resist the urge to disclose all the results of later analysis in the middle of the story, which may provide a slightly less clear view of the disaster, but which makes the telling far more compelling. I usually read non-fiction more slowly than fiction, but Fukushima dragged me in. I found myself grabbing moments to read just another few pages. Unfortunately, this is only about half the book. The other half is a mix of other things that won't have as broad of appeal: an analysis of the challenges of US nuclear regulation, a history of the US nuclear power industry, and a presentation of the authors' opinions about the best path forward for regulation of nuclear power in the US. Since I'm a US citizen and resident with an interest in both nuclear power and regulation of nuclear power in my country, I found this interesting, if not as engrossing as the rest of the book. But it felt a bit oddly tacked on, and I think it's a stretch to say that it's part of the story of Fukushima. The authors try to draw that link by presenting the Japanese nuclear power industry as heavily influenced by their US counterparts, and their regulatory problems as similar to the problems in the US, but there is nowhere near enough detail about Japanese regulatory practices here to support that conclusion. I think the largest weakness, and the most obvious gap, in this book is the lack of detailed analysis of the history and players in the Japanese nuclear regulatory environment. This is an odd miss. If one is concerned about regulatory inadequacy, Japanese government policy is far more obviously part of the story of Fukushima than US policy. I can only speculate that the authors had inside sources for the US policy discussions but not for the Japanese policy discussions (and, sadly, fall back on painting with a rather broad brush and making unsupported generalizations about Japanese regulatory approaches in a few spots). The result feels like two partly-unrelated books stacked and partly shuffled together. So, there are parts of Fukushima that are rather disappointing, particularly for non-US readers. But I still recommend it as a great detailed history of the actual incident and a summary of what we now think happened. That summary is unfortunately sketchy and still very unclear, but I don't think that's the fault of the authors. The inside of a nuclear power plant during a meltdown is a very difficult environment to measure or analyze, and there's a lot of data that we will probably never have. Some details may never be known. But what we do know, and how that knowledge unfolded, is told very well. This is the only book-length treatment on Fukushima I've read, so I can't compare it against other books on the same topic. But it satisfied my curiousity nicely. If you have a similar curiosity, I recommend this book to your attention, although be aware of its approach and its US-centric analysis going in so that you're not surprised by a mismatch of expectations. Rating: 8 out of 10

23 March 2015

Carl Chenet: Unverified backups are useless. Automatize the controls!

Follow me on Identi.ca or Twitter or Diaspora*diaspora-banner Unverified backups are useless, every sysadmins know that. But manually verifying a backup means wasting time and resources. Moreover it s boring. You should automatize it!
charlot

Charlie Chaplin Modern Times

Backup Checker is a command line software developed in Python 3.4 on GitHub (stars appreciated :) ) allowing users to verify the integrity of archives (tar, gz, bz2, lzma, zip, tree of files) and the state of the files inside an archive in order to find corruptions or intentional of accidental changes of states or removal of files inside an archive.
github-logo

Backup Checker on github

The new feature of the latest version 1.4 is the control of outdated archives with the new outdated parameter. Lots of data are outdated quite fast, because they are dependent of other data, or because they are only useful in a specific context.
Hey, this database dump is 6 months old, it s useless today!
Backup Checker now controls the expiration duration and triggers a warning if the given duration starting from the last modification of the archive (mtime) is expired. Short examples of the warning: WARNING:root:/backups/backups-12022015.tar.gz is outdated. Was good until 01/03/15 00:00:00 now 22/03/15 21:38:20 You won t be surprized any more by outdated useless data in your backups. Backup Checker also offers lots of other controls. Check the features list! Installing Backup Checker Backup Checker is available from PyPI using the following command:
# pip3.4 install backupchecker
It s also available for your Debian Squeeze or Debian Wheezy. Check how to get it for your specific distributions. weneedyou What about you? How and what for do you use Backup Checker? We would be happy to get your feedbacks. The project cares about our users and the outdated feature was a awesome idea in a feature request by one of the Backup Checker user, thanks Laurent!

Next.

Previous.