Note: In this post I mention some problems and ask questions (to myself, like thinking aloud ). The goal is not to get answers to those questions (I suppose that I will find them soon or later in the internet, manuals and so), but to show the kind of problems and questions that arise in my selfhosting adventures, which I suppose are common to other people trying to administer a home server with some web services.
Am I an
userop? Well I m something in the middle of (GNU/Linux) user and sysadmin: I have studied computer technical engineering but most of my experience has been in helpdesk, providing support for Windows users. I m running Debian in some LAMP boxes at work (without GUI) since 2008 or so, and in my desktops (with GUI) since 2010. I don t code nor package, but I don t mind trying to read code and understand it (or not). I know a bit of C, a bit of Python, of PHP, and enough Perl to open a Perl file and close it after two minutes, understanding that it s great, but too much for me
:) I translate software, so I m not scared to clone a repository, edit files, commit or submit a patch. I m not scared of compiling a program (except if it s an Android app: I try to avoid setting up the development environment just to try some translation that I made but I built my Puma before it was the binary available for download or in F-Droid).
In conclusion, I feel more like a GNU/Linux power user than a sysadmin . Sometimes just a user or even a newbie (for example, I don t know very well the Unix/Linux folder tree where are the wallpapers stored? Does it depend on the desktop that I use?).
Anyway. I won t stop my free software + free networks digital life because I don t know many things. I bought a small server for home last September, and I wanted to try to selfhost some services, for me and for my family. I want to be a home sysadmin or something like that, so I joined the
userops mailing list :)
Here you have my experiences on selfhosting/being an userop until now.
Mail
I even didn t try to setup my mail server, because many people say it s a pain (although nice articles were published about how to do it, for example
this series in ArsTechnica) and I need a static IP which is 14 /month more to my ISP, and
Gandi, the place where I rented my domain name, provides mail, and they use Debian and Roundcube, and sponsor Debian too, so I decided to trust on them.
So this is my strategy now, to try to keep mail under my control:
- Trust my domain provider.
- Backup my mail and keep local copies, removing sensible stuff from the server.
- Use and spread the word about GPG encryption.
- Try not to send photos or videos by mail, just send the link to my MediaGoblin instance (see below).
MediaGoblin
I ve setup two
MediaGoblin instances (yes, two!). I managed to do it in Debian 7 stable (I think NodeJS npm was not needed then), but soon later I upgraded to Jessie so now it s even better.
I installed Nginx and PostgreSQL via apt, to use them for both instances (and probably some more software later).
One instance is public, and I use a Debian user, a PostgreSQL database, and it s running in
http://media.larjona.net
I have requested an SSL cert to Gandi but I still didn t deployed it (lazy LArjona!!).
The other instance is private, for family photos. I didn t know very well how much of my existing setup could reuse and how to keep both instances in case of downtimes or attack I know more or less the concept or chroot but I don t know how to deploy it in my machine. So I decided to use another Debian user, another PostgreSQL database, deploy MediaGoblin in a different folder, and create another virtual server in my Nginx to serve it. I managed to setup that virtual server to http-authenticate and to serve content via a different port, and use a self-signed SSL certificate (it s only for family, so it does not matter). I created another (unprivileged) Debian user with a password for the nginx authentication, and gave my family the URL in the form
https://mediaprivate.larjona.net:PortNumber and the user and password (
mediaprivate is a string, and
PortNumber is a number). I think they don t use the instance too much, but at least I upload photos there from time to time and email the link instead of emailing the photos themselves (they don t use GPG either ).
Upgrades
I upgraded MediaGoblin from 0.7.1 to 0.8.0 successfully,
I sent a report about how I did it to the mailing list. First I upgraded the public instance, when I figure out the process, I upgraded the second instance to test my instructions, and then, I sent the report with the instructions to the mailing list.
Static site and LimeSurvey: the power of free software (with instructions)
I wanted to act as a mirror of
floss2013.libresoft.es and
surveys.libresoft.es since they suffer a downtime and I participated in that project (not in the sysadmin part, but in the research and content creation).
The static site floss2013.libresoft.es offered a zip with the whole website tree (since the website was licensed as AGPL), and I had access to the git repo holding the development copy of the website. So I just cloned the repository and setup another nginx virtual server in my machine, and tuned my DNS zone in Gandi website to serve
floss2013.larjona.net from home. 10 minutes setup YAY! #inGitWeTrust #FreeSoftwareFTW
:)
For
surveys.larjona.net I had to install a LimeSurvey instance. I knew how to do it because we use LimeSurvey at work, but at home I had Nginx instead of Apache, and PostgreSQL instead of MySQL. And no PHP I searched about how to install PHP in Nginx (I can use apt-get, nice!) and how to install LimeSurvey with Nginx and PostgreSQL (I had documentation about that, so I followed, and it worked).
For making available the data (one survey and its results, so people can login as visitor to query and get statistics), I downloaded the LimeSurvey export dataset that we were providing in the static website, followed the
replication instructions (hey, I wrote them!), and they worked #oleole! (
And here, dear researchers, gets demonstrated that free software and free culture really empower your research and help spreading your results).
Etherpad: not so easy, it seems!
I m trying to install Etherpad-Lite, but I m suffering a bit. I think I did everything ok according to some guides but I get Bad Gateway and these kind of errors when trying to browse with Lynx in the host:
[error] 3615#0: *24 upstream timed out (110: Connection timed out)
while reading response header from upstream,
client: 127.0.0.1,
server: pad.larjona.net,
request: "GET / HTTP/1.0",
upstream: "http://127.0.0.1:9001/",
host: "pad.larjona.net"
2015/04/17 20:52:56 [error] 3615#0: *24 connect() failed
(111: Connection refused) while connecting to upstream,
client: 127.0.0.1,
server: pad.larjona.net,
request: "GET / HTTP/1.0",
upstream: "http://[::1]:9001/",
host: "pad.larjona.net"
I m not sure if I need to open some port in iptables, my router, or change my nginx configuration because the guides assume you re only serving one website in the port 80 (and I have several of them, now ), or what I ve spent three chunks of time (maybe ~2h each?) on this, in different days, and couldn t figure it out, so I decided to round-robin in my TODO list.
Userops thoughts
Debian brings peace of mind (for me)
On one side, maintaining a Debian box it s quite easy, and the more software that it s packaged, the less time that I spend installing or upgrading. I like being in stable, I m in Jessie now (I migrated when it was frozen), but I ll stay in stable as much as I can.
I like that I can use the software that I installed via apt-get for several services (nginx, PostgreSQL ). About the software that is not packaged (MediaGoblin, LimeSurvey, EtherPad, maybe others later), I wonder how dependencies and updates are handled. And maybe (probably) I have installed some components several times, one for each service (this sounds like a Windows box #grr).
For example MediaGoblin uses PyPump. PyPump 0.5 is packaged in Debian Jessie. MediaGoblin uses PyPump 0.7+. What if PyPump 0.7+ gets, let s say, into Jessie-backports? Can I benefit from that?
I know that MediaGoblin upgrade instructions includes upgrading the dependencies, but what about some security patch in one dependency? Should I upgrade the pip modules periodically? How to know if some upgrade is recommended because patches a vulnerability, or it s just new features (and maybe breaking my setup)?
This kind of things are the peace of mind that Debian packaging brings to me: when some piece of software is packaged, I know maybe I need to care about proper setup and configuration, but later, it s kind-of-easy to maintain (since the Debian maintainers care about the rest). I don t mind about cloning a repo and compiling, I mind about later, or coexistance with other program/services. I trust in the MediaGoblin community and I m an active member (I m not developer, but hang on IRC, follow the mailing list, etc) but for example I don t know anything about the EtherPad project. And I don t feel like joining the community (I m already an active member in Debian, MediaGoblin, F-Droid, Pump.io, translator of LimeSurvey and many other small apps that I use, and in the future will use more services, like OwnCloud, XMPP ), joining the community of each software that I use is becoming not sustainable :s
Free software is more than software
I follow the userop mailing list, and it s becoming very technical. I mostly understand the problems (which are similar to the problems that I face: how to isolate different services, how to easily-configure them, how to make them installable by average user ) But I don t understand most of the solutions proposed, and I think that probably we need technical solutions, but in the meanwhile, some issues can be addressed not with software, but with other means: good documentation, community support, translations, beta-testers
This is my conclusion until now. When a project is well documented, I think I can find my way to selfhost no matter if the software is packaged (or contained ) or not. MediaGoblin, and LimeSurvey, are well documented, and the user support channels are very responsive.
I find lots of instructions that assume that you will use a whole machine for their service (and not for other things). And lots of documentation for the LAMP stack, but not for Nginx + PostgreSQL and Node instead of PHP So, for each particularity of my setup, I search the internet and try to pick good sources to help me do what I wanted to do.
I m kind of privileged
Some elements, not software related, to take into account as pre-requisites for succeed selfhosting services:
- I knew what to search.
- I knew which sites to visit from the results (arch wiki, debian wiki, stack overflow, etc: some of them were not the Top1 in the results).
- I had time to read several sources and make my mind about what to do and how.
- I can read, understand, and write in English.
- I have no fear about my broken English.
- I have no impostor syndrome.
- I felt welcome in the FLOSS communities where I hanged out.
These aspects are not present in a lot of people. If I look around to the computer users that I know (mostly Windows+Android, some GNU/Linux users, some Mac OSX users, some iOS users), I find that they search things like X does not work or they cannot write a proper search query in English. Or they trust some random person writing a recipe in their blog, without trying first to understand what the recipe does. Other people just say I m not a professional sysadmin, I ll just do what everybody does (aka use Google services or whatever). What if I try and I don t succeed? . Things like that.
We may need some technical solutions (and hackers are thinking about that, and working on that). But I feel that we need (more) a huge group of beta-testers, dogfooding people, aventurers that try the half-cooked solutions and provide successful and unsuccessful experiences, to guide the research and make software technologies advance. I m not sure if I am an userop, but I feel part of that vanguard force , I want to be part of the future of free software and free networks.
Comments?
You can comment about this post on
this pump.io thread.Filed under:
My experiences and opinion Tagged:
Communities,
Contributing to libre software,
Debian,
Developer motivations,
English,
free networks,
Free Software,
Freedom,
innovation,
MediaGoblin,
Moving into free software,
Project Management,
selfhosting,
sysadmin 
Thursday was our big day: After more than 2.5 years of working in the
hidden, ups and downs, Frank
Hofmann and myself were able to announce the availability of our
book project Debian
Package Management under a free license (Creative Commons Attribution ShareAlike 4.0 International
License, short CC BY-SA 4.0 ) during a Lightning Talk at DebConf15 in Heidelberg.
This became possible because we found Onyx Neon, a publishing company
which is specialised on books with contents under free licenses. Its
founder does not only have a faible for Perl but also for Debian.
(Since the question already came up: We also thought about
self-publishing, e.g. via Lulu or Epubli and it would have been our fallback solution
, but we prefer the professionalism and services of a real
publisher. I m though happy to share what I found out about
self-publishing in the past few months.)
The source code of the book is written in the AsciiDoc format and available on
GitHub.
The book is still work in progress. But if you want, you can
already build an e-book out of the publically available source code:
Following the tradition of 
Found at the 
Just came back from a wonderful time hiking with my daughter at the finale 
A couple of months ago, I was invited to give the starting course for the 
I recently obtained the newest Dell's Ubuntu developer offering, XPS 13 (2015, model 9343). I opted in for FullHD non-touch display, mostly because of better battery life, the actual no need for higher resolution, and matte screen which is great outside. Touch would have been "nice-to-have", but in my work I don't really need it.
What happened about the 
Who are you?
I am 
Since 2009, I am persuing a personal programming project.
As I am not a professional programmer, I have spent quite a lot of that
time exploring options. I have thrown out about three or four prototype
implementations already. My last implementation seems to contain enough
accumulated wisdom to be actually useful. I am far from finished, but the path
I am walking now seems relatively sound.
So, what is this project about? I have set myself a rather ambitious
goal: I am trying to implement a two-way bridge between visual music notation
and 
So, I also got myself a new toy. My current ThinkPad is a bit
ancient, but still sturdy. It's an X201s from 2010 (brought
refurbished), and it's still working pretty fine, but eh, I
couldn't resist.
The X230 was nice, but didn't have a large resolution screen
(1366 768). The X240 brought a full HD (1920 1080) IPS screen,
but lost the hardware trackpoint buttons. Finally, the X250 brings
back the buttons, still have a nice screen (not qHD or some other
trendy resolutions, but still FHD and IPS). And on top of that, it
comes with Broadwell, so that means I get smap.
It runs mostly fine out of the box on Debian sid, but for full
support some tuning is needed. I've setup a 
A few weeks ago I went to see Blade Runner: The Final Cut in a one-off
showing at the 
There s a lot of discussion / moaning /arguing at this time, so I thought I d post something about how LAVA got into Debian Jessie, the work involved and the lessons I ve learnt. Hopefully, it will help someone avoid the disappointment of having their package missing the migration into a future stable release. This was going to be a talk at the Minidebconf-uk in Cambridge but I decided to put this out as a permanent blog entry in the hope that it will be a useful reference for the future, not just Jessie.
Context
LAVA relies on a number of dependencies which were at the time all this started NEW to Debian as well as many others already in Debian. I d been running LAVA using packages on my own system for a few months before the packages were ready for use on the main servers (I never actually installed LAVA using the old virtualenv method on my own systems, except in a VM). I did do quite a lot of this on my own but I also had a team supporting the effort and valuing the benefits of moving to a packaged system.
At the time, LAVA was based on Ubuntu (12.04 LTS Precise Pangolin) and a new Ubuntu LTS was close (Trusty Tahr 14.04) but I started work on this in 2013. By the time my packages were ready for general usage, it was winter 2013 and much too close to get anything into Ubuntu in time for Trusty. So I started a local repo using space provided by Linaro. At the same time, I started uploading the dependencies to Debian. json-schema-validator, django-testscenarios and others arrived in April and May 2014. (Trusty was released in April). LAVA arrived in NEW in May, being accepted into unstable at the end of June. LAVA arrived in testing for the first time in July 2014.
Upstream development continued apace and a regular monthly upload, with some hotfixes in between, continued until close to the freeze.
At this point, note that although upstream is a medium sized team, the Debian packaging also has a team but all the uploads were made by me. I planned ahead. I knew that I would be going to Macau for Linaro Connect in February a critical stage in the finalisation of the packages and the migration of existing instances from the old methods. I knew that I would be on vacation from August through to the end of September 2014 including at least two weeks with absolutely no connectivity of any kind.
Right at this time, Django1.7 arrived in experimental with the intent to go into unstable and hence into Jessie. This was a headache for me, I initially sought to delay the migration until after Jessie. However, we discussed it upstream, allocated time within the busy schedule and also sought help from within Debian with the RFH tag. Rapha l Hertzog contributed patches for django1.7 support and we worked on those patches upstream, once I was back from vacation. (The final week of my vacation was a work conference, so we had everyone together at one hacking table.)
Still there was more to do, the django1.7 patches allowed the unit tests to complete but broke other parts of the lava-server package and needed subsequent tweaks and fixes.
Even with all this, the auto-removal from testing for packages affected by RC bugs in their dependencies became very important to monitor (it still is). It would be useful if some packages had less complex dependency chains (I m looking at you, uwsgi) as the auto-removal also covers build-depends. This led to some more headaches with libmatheval. I m not good with functional programming languages, I did have some exposure to Scheme when working on Gnucash upstream but it wasn t pleasant. The thought of fixing a scheme problem in the test suite of libmatheval was daunting. Again though, asking for help, I found people in the upstream team who wanted to refresh their use of scheme and were able to help out. The fix migrated into testing in October.
Just for added complications, lava-server gained a few RC bugs of it s own during that time too fixed upstream but awkward nonetheless.
Achievement unlocked
So that s how a complex package like lava-server gets into stable. With a lot of help. The main problem with top-level packages like this is the sheer weight of the dependency chain. Something seemingly unrelated (like libmatheval) can seriously derail the migrations. The package doesn t use the matheval support provided by uwsgi. The bug in matheval wasn t in the parts of matheval used by uwsgi. It wasn t in a language I am at all comfortable in fixing but it s my name on the changelog of the NMU. That happened because I asked for help. OK, when django1.7 was scheduled to arrive in Debian unstable and I knew that lava was not ready, I reacted out of fear and anxiety. However, I sought help, help was provided and that help was enough to get upstream to a point where the side-effects of the required changes could be fixed.
Maintaining a top-level package in Debian is becoming more like maintaining a core package in Debian and that is a good thing. When your package has a lot of dependencies, those dependencies become part of the maintenance workload of your package. It doesn t matter if those are install time dependencies, build dependencies or reverse dependencies. It doesn t actually matter if the issues in those packages are in languages you would personally wish to be expunged from the archive. It becomes your problem but not yours alone.
Debian has a lot of flames right now and Enrico encouraged us to look at what else is actually happening in Debian besides those arguments. Well, on top of all this with lava, I also did what I could to help the arm64 port along and I m very happy that this has been accepted into Jessie as an official release architecture. That s a much bigger story than LAVA yet LAVA was and remains instrumental in how arm64 gained the support in the kernel and various upstreams which allowed patches to be accepted and fixes to be incorporated into Debian packages.
So a roll call of helpers who may otherwise not have been recognised via changelogs, in no particular order:
I first want to clarify for the people not intimately involved with Debian that 
