Search Results: "tar"

1 September 2016

Jamie McClelland: Trusted Mobile Device: How hard could it be?

I bought a new phone. After my experiences with signal and the helpful comments readers gave regarding the ability to run android and signal without Google Play using microg I thought I would give it a shot. Since microg reports that signature spoofing is required and comes out-of-the-box with omnirom I thought I'd aim for installing omnirom's version of Android 6 (marshmallow) after years of using cyanomgenmod's version of Android. The Nexus line of phones seemed well-supported by omnirom in particular (and the alternative ROM community in general) so I bought a Nexus 5x. I carefully followed the directions for installing omnirom however when it came time to boot into omnirom, I just got the boot sequence animation over and over again. Frustrated, I decided to go back to cyanogenmod and see if I could use one of the microg recommended methods for getting signature spoofing to work. The easiest seemed to be Needle by moosd but alas no such luck with Marshmallow. Someone else forked the code and might fix it one day. I then spent too much time trying to understand what xposed is before I gave up understanding it and just tried to install it (woops, looks like the installer page is out of date so instead I followed sketchy instructions from a forum thread). Well, to make a long story short it resulted in a boot loop. So, I decided to return to omnirom. After reading some vague references to omnirom and supersu, I decided to flash both of them together and voila, it worked! Next, I decided to enable full disk encryption. Not so fast. After clicking through the screens and hitting the final confirmation, my phone rebooted and spent the next 5 hours showing me the omnirom boot animation. Somehow, powering down and starting again resulted in a working machine, but no disk encryption. After much web searching, guessing and trial and error, I fixed the problem by clicking on the SuperSU option to "Full unroot" the device (I pressed "no" when prompted to attempt to restore stock image). Then I rebooted and followed the directions to encrypt the device. And it worked! Hooray! I had to reboot and re-flash the supersu to regain su privileges. All was great. The first root action I decided to take was to install the cryptfs program from f-droid because using the same password to decrypt your device as you use to unlock the screen seems either tedious or insecure. That process didn't work so well. I got a message saying: use this command from a root shell before you reboot: vdc cryptfs changepw <password>. I followed the advice, carefully typing in my 12 character password which includes numbers and letters. Then, I happily did what I expected to be my last reboot when, to my horror, I was prompted to decrypt my disk with ... a numeric-only keypad. That wasn't going to work. At this point I had already spent 5 days and about 8 hours on this project. Sigh. So, I started over. Guess what? It only took me 25 minutes but, it seems that cryptfs is broken. Even with a numeric password it fails. Ok, I guess I need a long pin to unlock my phone. This time it only took my 15 minutes to wipe and re-install everything. There are only two positive things I can think of: p.s. I haven't even tried to install microg yet... which was the whole point.

31 August 2016

Enrico Zini: Links for September 2016

A Few Useful Mental Tools from Richard Feynman [archive]
These tricks show Feynman taking the method of thought he learned in pure science and applying it to the more mundane topics most of us have to deal with every day.
Pasta [archive]
A comprehensive introduction to pasta, to keep at hand in case I meet someone who has little familiarity with it.
MPTP: One Designer Drug and Serendipity [archive]
Abstract: Through an unlikely series of coincidences and fortunate accidents, the development of Parkinson s disease in several illicit drug users was traced to their use of a meperidine analog contaminated with 1-methyl-4-phenyl-1,2,3,6-tetrahydropyridine (MPTP). The discovery of a chemical capable of producing animal models of the disease has revitalized research efforts and resulted in important new information. The serendipitous finding also prompted consideration of what changes seem advisable if designer drugs are to be dealt with more efficaciously.
The Debunking Handbook: now freely available for download
The Debunking Handbook, a guide to debunking misinformation, is now freely available to download. Although there is a great deal of psychological research on misinformation, there's no summary of the literature that offers practical guidelines on the most effective ways of reducing the influence of myths.
Faulty neon light jams radio appliances [archive]
An apparent interference source began plaguing wireless vehicle key fobs, cell phones, and other wireless electronics. Key fob owners found they could not open or start their vehicles remotely until their vehicles were towed at least a block away, nor were they able to call for help on their cell phones when problems occurred
Calvin & Muad'Dib
Calvin & Hobbes with text taken from Frank Herbert's Dune. It's been around since 2013 and I consistently found it moving and deep.
When Birds Attack - Bike Helmet Hacks [archive]
Australian magpies attacking cyclists has propted several creative adaptations, including attaching an afro wig to the bike helmet.

Chris Lamb: Free software activities in August 2016

Here is my monthly update covering what I have been doing in the free software world (previously):

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most Linux distributions provide binary (or "compiled") packages to end users. The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced either maliciously and accidentally during this compilation process by promising identical binary packages are always generated from a given source.

Diffoscope diffoscope is our "diff on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them:
  • Added a command-line interface to the try.diffoscope.org web service.
  • Added a JSON comparator.
  • In the HTML output, highlight lines when hovering to make it easier to visually track.
  • Ensure that we pass str types to our Difference class, otherwise we can't be sure we can render them later.
  • Testsuite improvements:
    • Generate test coverage reports.
    • Add tests for Haskell and GitIndex comparators.
    • Completely refactored all of the comparator tests, extracting out commonly-used routines.
    • Confirm rendering of text and HTML presenters when checking non-existing files.
    • Dropped a squashfs test as it was simply too unreliable and/or has too many requirements to satisfy.
  • A large number of miscellaneous cleanups, including:
    • Reworking the comparator setup/preference internals by dynamically importing classes via a single list.
    • Split exceptions out into dedicated diffoscope.exc module.
    • Tidying the PROVIDERS dict in diffoscope/__init__.py.
    • Use html.escape over xml.sax.saxutils.escape, cgi.escape, etc.
    • Removing hard-coding of manual page targets names in debian/rules.
    • Specify all string format arguments as logging function parameters, not using interpolation.
    • Tidying imports, correcting indentation levels and drop unnecessary whitespace.

disorderfs disorderfs is our FUSE filesystem that deliberately introduces nondeterminism in system calls such as readdir(3).
  • Added a testsuite to prevent regressions. (f124965)
  • Added a --sort-dirents=yes no option for forcing deterministic ordering. (2aae325)

Other
  • Improved strip-nondeterminism, our tool to remove specific nondeterministic information after a build:
    • Match more styles of Java .properties files.
    • Remove hyphen from "non-determinism" and "non-deterministic" throughout package for consistency.
  • Improvements to our testing infrastucture:
    • Improve the top-level navigation so that we can always get back to "home" of a package.
    • Give expandable elements cursor: pointer CSS styling to highlight they are clickable.
    • Drop various trailing underlined whitespaces after links.
    • Explicitly log that build was successful or not.
    • Various code-quality improvements, including prefering str.format over concatentation.
  • Miscellaneous updates to our filter-packages internal tool:
    • Add --random=N and --url options.
    • Add support for --show=comments.
    • Correct ordering so that --show-version runs after --filter-ftbfs.
    • Rename --show-ftbfs to --filter-ftbfs and --show-version to --show=version.
  • Created a proof-of-concept reproducible-utils package to contain commonly-used snippets aimed at developers wishing to make their packages reproducible.


I also submitted 92 patches to fix specific reproducibility issues in advi, amora-server, apt-cacher-ng, ara, argyll, audiotools, bam, bedtools, binutils-m68hc1x, botan1.10, broccoli, congress, cookiecutter, dacs, dapl, dateutils, ddd, dicom3tools, dispcalgui, dnssec-trigger, echoping, eekboek, emacspeak, eyed3, fdroidserver, flashrom, fntsample, forkstat, gkrellm, gkrellm, gnunet-gtk, handbrake, hardinfo, ircd-irc2, ircd-ircu, jack-audio-connection-kit, jpy, kxmlgui, libbson, libdc0, libdevel-cover-perl, libfm, libpam-ldap, libquvi, librep, lilyterm, mozvoikko, mp4h, mp4v2, myghty, n2n, nagios-nrpe, nikwi, nmh, nsnake, openhackware, pd-pdstring, phpab, phpdox, phpldapadmin, pixelmed-codec, pleiades, pybit, pygtksourceview, pyicu, python-attrs, python-gflags, quvi, radare2, rc, rest2web, roaraudio, rt-extension-customfieldsonupdate, ruby-compass, ruby-pg, sheepdog, tf5, ttf-tiresias, ttf-tiresias, tuxpaint, tuxpaint-config, twitter-bootstrap3, udpcast, uhub, valknut, varnish, vips, vit, wims, winswitch, wmweather+ & xshisen.

Debian GNU/Linux
Debian LTS

This month I have been paid to work 15 hours on Debian Long Term Support (LTS). In that time I did the following:
  • "Frontdesk" duties, triaging CVEs, etc.
  • Authored the patch & issued DLA 596-1 for extplorer, a web-based file manager, fixing an archive traversal exploit.
  • Issued DLA 598-1 for suckless-tools, fixing a segmentation fault in the slock screen locking tool.
  • Issued DLA 599-1 for cracklib2, a pro-active password checker library, fixing a stack-based buffer overflow when parsing large GECOS fields.
  • Improved the find-work internal tool adding optional colour highlighting and migrating it to Python 3.
  • Wrote an lts-missing-uploads tool to find mistakes where there was no correponding package in the archive after an announcement.
  • Added optional colour highlighting to the lts-cve-triage tool.
Uploads
  • redis 2:3.2.3-1 New upstream release, move to the DEP-5 debian/copyright format, ensure that we are running as root in LSB initscripts and add a README.Source regarding our local copies of redis.conf and sentinel.conf.
  • python-django:
    • 1:1.10-1 New upstream release.
    • 1:1.10-2 Fix test failures due to mishandled upstream translation updates.

  • gunicorn:
    • 19.6.0-2 Reload logrotate in the postrotate action to avoid processes writing to the old files and move to DEP-5 debian/copyright format.
    • 19.6.0-3 Drop our /usr/sbin/gunicorn ,3 -debian and related Debian-specific machinery to be more like upstream.
    • 19.6.0-4 Drop "template" systemd .service files and point towards examples and documentation instead.

  • adminer:
    • 4.2.5-1 Take over package maintenance, completely overhauling the packaging with a new upstream version, move to virtual-mysql-server to support MariaDB, updating package names of dependencies and fix the outdated Apache configuration.
    • 4.2.5-2 Correct the php5 package names.

FTP Team As a Debian FTP assistant I ACCEPTed 90 packages: android-platform-external-jsilver, android-platform-frameworks-data-binding, camlpdf, consolation, dfwinreg, diffoscope, django-restricted-resource, django-testproject, django-testscenarios, gitlab-ci-multi-runner, gnome-shell-extension-taskbar, golang-github-flynn-archive-go-shlex, golang-github-jamesclonk-vultr, golang-github-weppos-dnsimple-go, golang-golang-x-time, google-android-ndk-installer, haskell-expiring-cache-map, haskell-hclip, haskell-hdbc-session, haskell-microlens-ghc, haskell-names-th, haskell-persistable-record, haskell-should-not-typecheck, haskell-soap, haskell-soap-tls, haskell-th-reify-compat, haskell-with-location, haskell-wreq, kbtin, libclipboard-perl, libgtk3-simplelist-perl, libjs-jquery-selectize.js, liblemon, libplack-middleware-header-perl, libreoffice, libreswan, libtest-deep-json-perl, libtest-timer-perl, linux, linux-signed, live-tasks, llvm-toolchain-3.8, llvm-toolchain-snapshot, lua-luv, lua-torch-image, lua-torch-nn, magic-wormhole, mini-buildd, ncbi-vdb, node-ast-util, node-es6-module-transpiler, node-es6-promise, node-inline-source-map, node-number-is-nan, node-object-assign, nvidia-graphics-drivers, openhft-chronicle-bytes, openhft-chronicle-core, openhft-chronicle-network, openhft-chronicle-threads, openhft-chronicle-wire, pycodestyle, python-aptly, python-atomicwrites, python-click-log, python-django-casclient, python-git-os-job, python-hypothesis, python-nosehtmloutput, python-overpy, python-parsel, python-prov, python-py, python-schema, python-tackerclient, python-tornado, pyvo, r-cran-cairo, r-cran-mi, r-cran-rcppgsl, r-cran-sem, ruby-curses, ruby-fog-rackspace, ruby-mixlib-archive, ruby-tzinfo-data, salt-formula-swift, scapy3k, self-destructing-cookies, trollius-redis & websploit.

Joey Hess: late summer

With days beginning to shorten toward fall, my house is in initial power saving mode. Particularly, the internet gateway is powered off overnight. Still running electric lights until bedtime, and still using the inverter and other power without much conservation during the day. Indeed, I had two laptops running cpu-melting keysafe benchmarks for much of today and one of them had to charge up from empty too. That's why the house power is a little low, at 11.0 volts now, despite over 30 amp-hours of power having been produced on this mostly clear day. (1 week average is 18.7 amp-hours) September/October is the tricky time where it's easy to fall off a battery depletion cliff and be stuck digging out for a long time. So time to start dusting off the conservation habits after summer's excess.
I think this is the first time I've mentioned any details of living off grid with a bare minimum of PV capacity in over 4 years. Solar has a lot of older posts about it, and I'm going to note down the typical milestones and events over the next 8 months.

30 August 2016

Daniel Stender: My work for Debian in August

Here's again a little list of my humble off-time contributions I'm happy to add to the large amount of work we're completing all together each month. Then there is one more "new in Debian" (meaning: "new in unstable") announcement. First, the uploads (a few of them are from July): New packages: Sponsored uploads: Requested resp. suggested for packaging: New in Debian: Lasagne (deep learning framework) Now that the mathematical expression compiler Theano is available in Debian, deep learning frameworks resp. toolkits which have been build on top of it can become available within Debian, too (like Blocks, mentioned before). Theano is an own general computing engine which has been developed with a focus on machine learning resp. neural networks, which features an own declarative tensor language. The toolkits which have build upon it vary in the way how much they abstract the bare features of Theano, if they are "thick" or "thin" so to say. When the abstraction gets higher you gain more end user convenience up to the level that you have the architectural components of neural networks available for combination like in a lego box, while the more complicated things which are going on "under the hood" (like how the networks are actually implemented) are hidden. The downside is, thick abstraction layers usually makes it difficult to implement novel features like custom layers or loss functions. So more experienced users and specialists might to seek out for the lower abstraction toolkits, where you have to think more in terms of Theano. I've got an initial package of Keras in experimental (1.0.7-1), it runs (only a Python 3 package is available so far) but needs some more work (e.g. building the documentation with mkdocs). Keras is a minimalistic, high modular DNN library inspired by Torch1. It has a clean, rather easy API for experimenting and fast prototyping. It can also run on top of Google's TensorFlow, and we're going to have it ready for that, too. Lasagne follows a different approach. It's, like Keras and Blocks, a Python library to create and train multi-layered artificial neural networks in/on Theano for applications like image recognition resp. classification, speech recognition, image caption generation or other purposes like style transfers from paintings to pictures2. It abstracts Theano as little as possible, and could be seen rather like an extension or an add-on than an abstraction3. Therefore, knowledge on how things are working in Theano would be needed to make full use out of this piece of software. With the new Debian package (0.1+git20160728.8b66737-1)4, the whole required software stack (the corresponding Theano package, NumPy, SciPy, a BLAS implementation, and the nividia-cuda-toolkit and NVIDIA kernel driver to carry out computations on the GPU5) could be installed most conveniently just by a single apt-get install python ,3 -lasagne command6. If wanted with the documentation package lasagne-doc for offline use (no running around on remote airports seeking for a WIFI spot), either in the Python 2 or the Python 3 branch, or both flavours altogether7. While others have to spend a whole weekend gathering, compiling and installing the needed libraries you can grab yourself a fresh cup of coffee. These are the advantages of a fully integrated system (sublime message, as always: desktop users switch to Linux!). When the installation of packages has completed, the MNIST example of Lasagne could be used for a quick check if the whole library stack works properly8: 
$ THEANO_FLAGS=device=gpu,floatX=float32 python /usr/share/doc/python-lasagne/examples/mnist.py mlp 5
Using gpu device 0: GeForce 940M (CNMeM is disabled, cuDNN 5005)
Loading data...
Downloading train-images-idx3-ubyte.gz
Downloading train-labels-idx1-ubyte.gz
Downloading t10k-images-idx3-ubyte.gz
Downloading t10k-labels-idx1-ubyte.gz
Building model and compiling functions...
Starting training...
Epoch 1 of 5 took 2.488s
  training loss:        1.217167
  validation loss:      0.407390
  validation accuracy:      88.79 %
Epoch 2 of 5 took 2.460s
  training loss:        0.568058
  validation loss:      0.306875
  validation accuracy:      91.31 %
The example on how to train a neural network on the MNIST database of handwritten digits is refined (it also provides --help) and explained in detail in the Tutorial section of the documentation in /usr/share/doc/lasagne-doc/html/. Very good starting points are also the IPython notebooks that are available from the tutorials by Eben Olson9 and Geoffrey French on the PyData London 201610. There you have Theano basics, examples for employing convolutional neural networks (CNN) and recurrent neural networks (RNN) for a range of different purposes, how to use pre-trained networks for image recognition, etc.
  1. For a quick comparison of Keras and Lasagne with other toolkits, see Alex Rubinsteyn's PyData NYC 2015 presentation on using LSTM (long short term memory) networks on varying length sequence data like Grimm's fairy tales (https://www.youtube.com/watch?v=E92jDCmJNek 27:30 sq.)
  2. https://github.com/Lasagne/Recipes/tree/master/examples/styletransfer
  3. Great introduction to Theano and Lasagne by Eben Olson on the PyData NYC 2015: https://www.youtube.com/watch?v=dtGhSE1PFh0
  4. The package is "freelancing" currently being in collab-maint, to set up a deep learning packaging team within Debian is in the stage of discussion.
  5. Only available for amd64 and ppc64el.
  6. You would need "testing" as package source in /etc/apt/sources.list to install it from the archive at the present time (I have that for years, but if Debian Testing could be advised as productive system is going to be discussed elsewhere), but it's coming up for Debian 9. The cuda-toolkit and pycuda are in the non-free section of the archive, thus non-free (mostly used in combination with contrib) must be added to main. Plus, it's a mere suggestion of the Theano packages to keep Theano in main, so --install-suggests is needed to pull it automatically with the same command, or this must be given explicitly.
  7. For dealing with Theano in Debian, see this previous blog posting
  8. Like suggested in the guideline From Zero to Lasagne on Ubuntu 14.04. cuDNN isn't available as official Debian package yet, but could be downloaded as a .deb package after registration at https://developer.nvidia.com/cudnn. It integrates well out of the box.
  9. https://github.com/ebenolson/pydata2015
  10. https://github.com/Britefury/deep-learning-tutorial-pydata2016, video: https://www.youtube.com/watch?v=DlNR1MrK4qE

Christoph Egger: DANE and DNSSEC Monitoring

At this year's FrOSCon I repeted my presentation on DNSSEC. In the audience, there was the suggestion of a lack of proper monitoring plugins for a DANE and DNSSEC infrastructure that was easily available. As I already had some personal tools around and some spare time to burn I've just started a repository with some useful tools. It's available on my website and has mirrors on Gitlab and Github. I intent to keep this repository up-to-date with my personal requirements (which also means adding a xmpp check soon) and am happy to take any contributions (either by mail or as "pull requests" on one of the two mirrors). It currently has smtp (both ssmtp and starttls) and https support as well as support for checking valid DNSSEC configuration of a zone. While working on it it turned out some things can be complicated. My language of choice was python3 (if only because the ssl library has improved since 2.7 a lot), however ldns and unbound in Debian lack python3 support in their bindings. This seems fixable as the source in Debian is buildable and useable with python3 so it just needs packaging adjustments. Funnily the ldns module, which is only needed for check_dnssec, in debian is currently buggy for python2 and python3 and ldns' python3 support is somewhat lacking so I spent several hours hunting SWIG problems.

Rhonda D'Vine: Thomas D

It's not often that an artist touches you deeply, but Thomas D managed to do so to the point of that I am (only half) jokingly saying that if there would be a church of Thomas D I would absolutely join it. His lyrics always did stand out for me in the context of the band I found about him, and the way he lives his life is definitely outstanding. And additionally there are these special songs that give so much and share a lot. I feel sorry for the people who don't understand German to be able to appreciate him. Here are three songs that I suggest you to listen to closely: Like always, enjoy!

/music permanent link Comments: 1 Flattr this

Petter Reinholdtsen: First draft Norwegian Bokm l edition of The Debian Administrator's Handbook now public

In April we started to work on a Norwegian Bokm l edition of the "open access" book on how to set up and administrate a Debian system. Today I am happy to report that the first draft is now publicly available. You can find it on get the Debian Administrator's Handbook page (under Other languages). The first eight chapters have a first draft translation, and we are working on proofreading the content. If you want to help out, please start contributing using the hosted weblate project page, and get in touch using the translators mailing list. Please also check out the instructions for contributors. A good way to contribute is to proofread the text and update weblate if you find errors. Our goal is still to make the Norwegian book available on paper as well as electronic form.

29 August 2016

David Moreno: Webhook Setup with Facebook::Messenger::Bot

The documentation for the Facebook Messenger API points out how to setup your initial bot webhook. I just committed a quick patch that would make it very easy to setup a quick script to get it done using the unreleased and still in progress Perl s Facebook::Messenger::Bot: 
use Facebook::Messenger::Bot;
use constant VERIFY_TOKEN => 'imsosecret';
my $bot = Facebook::Messenger::Bot->new(); # no config specified!
$bot->expect_verify_token( VERIFY_TOKEN );
$bot->spin();
This should get you sorted. What endpoint would that be, though? Well that depends on how you re giving Facebook access to your Plack s .psgi application.

Gergely Nagy: Recruitment mistakes: part 3

It has been a while that I have been contacted by a recruiter, and the last few ones were fairly decent conversations, where they made an effort to research me first, and even if they did not get everything right, they still listened, and we had a productive talk. But four days ago, I had another recruiter reach out to me, from a company I know oh so well: one I ranted about before: Google. Apparently, their recruiters still do carpet-bombing style outreach. My first thought was "what took them so long?" - it has been five years since my last contact with a Google recruiter. I almost started missing them. Almost. To think that Google is now powerful enough to read my mind, is scary. Yet, I believe, this is not the case; rather, it's just another embarrassing mistake.To make my case, let me quote the full e-mail I was sent, with the name of the sender redacted, and my comments - which I'm also sending to the recruiter:
Hi Gergely,
Hi!
How are you? Hope you're well.
Thank you, I'm fine, just back from vacation, and I was thrilled to read your e-mail. Although, I did find it surprising too, and considering past events, I thought it to be spam first.
My name is XXX and I am recruiting for the Google Engineering team. I have just found your details on Github...
I'm happy that you found me through GitHub, but I'm curious why you mailed me at my debian.org address then? That address is not on my GitHub profile, and even though I have some code signed with that address, that's not what I use normally. My GitHub profile also links to a page I created especially for recruiters, which I don't think you have read - but more on that below.
...and your experience with software development combined with your open source contributions is particularly relevant to Google
Which part of my recent contributions are relevant to Google? For the past few months, the vast majority (over 90%) of my open source work has been on keyboard firmware. If Google is developing a keyboard, then this may be relevant, otherwise, I find it doubtful.Some of my past OSS contributions may be more relevant, but that's in the past, and it would take some digging to see those from GitHub. And if you did that kind of digging, you would have found the page on my site for recruiters, and would not have e-mailed me at my Debian address, either.
We are always interested in talking to top engineers with your mix of skills and I was wondering if you are at all open to exploring roles with Google in EMEA.
To make things short, my stance is the same as it was five years ago, when I wrote - and I quote - "So, here's a public request: do not email me ever again about job opportunities within Google. I do not wish to work for Google. Not now, not tomorrow, not ever."This still stands. Please do not ever e-mail me about job opportunities within Google. I do not wish to work for Google, not now, not tomorrow, not five years from now, not ever. This will not change. Many things may change, this however, will not.But even if I ignore this, let me ask you mix of skills are you exactly interested in? Keyboard firmware hacking, mixed with Emacs Lisp, some Clojure, and hacking on Hy (which, if you have explored my GitHub profile, will surely know) from time to time? Or is it my Debian Developer hat that got your interest? If so, why didn't you say so? Not that it would change anything, but I'm curious.Is it my participation in 24pullrequests? Or my participation in GSoC in years past?Nevertheless, my list of conditions I mentioned above, apply. Is Google able to fulfill all my requirements, and the preferences too? Last time I heard, working for Google required one to relocate, which I clearly said on that page, I'm not willing to do.
The teams I recruit for are responsible for Google's planet-scale systems. Just to give you more of an idea, some of the projects we work on that might be of interest to you would include:
  • MapReduce
  • App Engine
  • Mesa
  • Maglev
And how would these be interesting to me, considering my recent OSS work? (Hint: none of them interest me, not a bit. Ok perhaps MapReduce, a little.)
I think you could be a great fit here, where you can develop a great career and at the same time you will be part of the most mission critical team, designing and developing systems which run Google Search, Gmail, YouTube, Google+ and many more as you can imagine.
My career is already developing fine, thank you, I do not need Google to do that. I am already working on things I consider important and interesting, if I wanted to change, I would. But I certainly would not consider a company which I had asked numerous times NOT to contact me about opportunities. If you can't respect this wish, and forget about this in mere five years, why would I trust you to keep any other promises you make now?
Thank you so much for your time and I look forward to hearing from you.
I wish you had spent as much time researching me - or even half that - as I have spent replying to you. I suppose this is not the reply you were expecting, or looking for, but this is the only one I'll ever give to anyone from Google.And here, at the end, if you read this far, I'm asking you, and everyone at Google to never contact me about job opportunities. I will not work for Google. Not now, not tomorrow, not five years from now, not ever. Please do not e-mail me again, and do not reply to this e-mail either. I'm not interested in neither apologies, nor promises that you won't contact me - just simply don't do it.Thank you.

Russell Coker: Monitoring of Monitoring

I was recently asked to get data from a computer that controlled security cameras after a crime had been committed. Due to the potential issues I refused to collect the computer and insisted on performing the work at the office of the company in question. Hard drives are vulnerable to damage from vibration and there is always a risk involved in moving hard drives or systems containing them. A hard drive with evidence of a crime provides additional potential complications. So I wanted to stay within view of the man who commissioned the work just so there could be no misunderstanding. The system had a single IDE disk. The fact that it had an IDE disk is an indication of the age of the system. One of the benefits of SATA over IDE is that swapping disks is much easier, SATA is designed for hot-swap and even systems that don t support hot-swap will have less risk of mechanical damage when changing disks if SATA is used instead of IDE. For an appliance type system where a disk might be expected to be changed by someone who s not a sysadmin SATA provides more benefits over IDE than for some other use cases. I connected the IDE disk to a USB-IDE device so I could read it from my laptop. But the disk just made repeated buzzing sounds while failing to spin up. This is an indication that the drive was probably experiencing stiction which is where the heads stick to the platters and the drive motor isn t strong enough to pull them off. In some cases hitting a drive will get it working again, but I m certainly not going to hit a drive that might be subject to legal action! I recommended referring the drive to a data recovery company. The probability of getting useful data from the disk in question seems very low. It could be that the drive had stiction for months or years. If the drive is recovered it might turn out to have data from years ago and not the recent data that is desired. It is possible that the drive only got stiction after being turned off, but I ll probably never know. Doing it Properly Ever since RAID was introduced there was never an excuse for having a single disk on it s own with important data. Linux Software RAID didn t support online rebuild when 10G was a large disk. But since the late 90 s it has worked well and there s no reason not to use it. The probability of a single IDE disk surviving long enough on it s own to capture useful security data is not particularly good. Even with 2 disks in a RAID-1 configuration there is a chance of data loss. Many years ago I ran a server at my parents house with 2 disks in a RAID-1 and both disks had errors on one hot summer. I wrote a program that s like ddrescue but which would read from the second disk if the first gave a read error and ended up not losing any important data AFAIK. BTRFS has some potential benefits for recovering from such situations but I don t recommend deploying BTRFS in embedded systems any time soon. Monitoring is a requirement for reliable operation. For desktop systems you can get by without specific monitoring, but that is because you are effectively relying on the user monitoring it themself. Since I started using mon (which is very easy to setup) I ve had it notify me of some problems with my laptop that I wouldn t have otherwise noticed. I think that ideally for desktop systems you should have monitoring of disk space, temperature, and certain critical daemons that need to be running but which the user wouldn t immediately notice if they crashed (such as cron and syslogd). There are some companies that provide 3G SIMs for embedded/IoT applications with rates that are significantly cheaper than any of the usual phone/tablet plans if you use small amounts of data or SMS. For a reliable CCTV system the best thing to do would be to have a monitoring contract and have the monitoring system trigger an event if there s a problem with the hard drive etc and also if the system fails to send a I m OK message for a certain period of time. I don t know if people are selling CCTV systems without monitoring to compete on price or if companies are cancelling monitoring contracts to save money. But whichever is happening it s significantly reducing the value derived from monitoring.
Related posts:
  1. Health and Status Monitoring via Smart Phone Health Monitoring Eric Topol gave an interesting TED talk about...
  2. Planning Servers for Failure Sometimes computers fail. If you run enough computers then you...
  3. Shelf-life of Hardware Recently I ve been having some problems with hardware dying. Having...

28 August 2016

Reproducible builds folks: Reproducible builds: week 70 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday August 21 and Saturday August 27 2016: GSoC and Outreachy updates Packages reviewed and fixed, and bugs filed Reviews of unreproducible packages 10 package reviews have been added and 6 have been updated this week, adding to our knowledge about identified issues. A large number of issue types have been updated: Weekly QA work 29 FTBFS bugs have been reported by: diffoscope development Holger also created another test job for diffoscope on jenkins.debian.net, so that now also all commits to branches other than master are being tested. strip-nondeterminism development strip-nondeterminism 0.023-1 was uploaded by Chris Lamb: 
 * Support Android .apk files with the JAR normalizer.
 * handlers/png.pm: Drop unused Archive::Zip import
 * Remove hyphen from non-determinism and non-deterministic.
 * javaproperties.pm: Match more styles of .properties and loosen filename matching.
 * Improve tests:
   - Make fixture runner generic to all normalizer types.
   - Replace (single) pearregistry test with a fixture.
   - Set a canonical time for fixture tests.
   - Add gzip testcase fixture.
   - Replace t/javadoc.t with fixture
   - Replace t/ar.t with a fixture.
   - t/javaproperties: move pom.properties and version.properties tests to fixtures
   - t/fixtures.t: move to using subtests
   - t/fixtures.t: Explicitly test that we can find a normalizer
   - t/fixtures.t: Don't run normalizer if we didn't find one.
strip-nondeterminism 0.023-2 uploaded by Mattia Rizzolo to allow stderr in autopkgtest. disorderfs development tests.reproducible-builds.org Debian: Somewhat related to reproducible builds there has been a first Debian jenkins team maintainance meeting on the #debian-qa IRC channel, to discuss current issues with the setup and to start the work of migrating jenkins.debian.net to jenkins.debian.org. The next meeting will take place on September 28th 2016 at 19 UTC. Misc. This week's edition was written by Chris Lamb and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

Hideki Yamane: please help to maintain net-snmp package

I stepped down from net-snmp package maintainers, and want someone to maintain it.

27 August 2016

Arturo Borrero Gonz lez: Why conntrackd in Debian is better with systemd


There has been some discussion [0] around my decision to drop sysvinit support in the conntrackd package in Debian [1] (version 1:1.4.4-2).

The rationale I used for such a movement was sent to the debian-devel [2] mailing list, and here it is:


Lots of people have talked to me with both support and disagreement with the change.

Before reading the rest of this blogpost, please note that I'm not interested in the 'systemd vs sysvinit' war, and starting now I will focus mainly on the subject of building a firewall cluster with netfilter technology and the reasons why I think sysvinit here is irrelevant.

I started working with firewall clusters by the time of Debian Squeeze. By then, I used only sysvinit, because it was the Debian default init system and because I did not dive so much in the internals of the firewall cluster itself.

By then, the conntrackd debian package included support for sysvinit by means of two files (the two files that I dropped in 1:1.4.4-2) :


Using both files, you could start/stop conntrackd, and nothing more (for example, a proper status check was not implemented).

The conntrackd daemon is used in HA firewall clusters to replicate connection states between nodes of the cluster, so flow states are known in all nodes and they can properly perform stateful firewalling.

When you build HA clusters, there are 2 basic states of the cluster you may check and adjust: failover and failback.

With conntrackd, the failover situation is straight forward: the other node has the state information already from the dying node.
The failback situation is different: depending on the configuration of both the cluster and conntrackd, the new node may ask for a complete synchronisation to the other node when it become alive again (at boot time, for example) . This is the case if you are building a multi-master firewall cluster (i.e: both nodes are seeing traffic and filtering at the same time).

Asking for a complete synchronisation is done by means of a new instance of conntrackd, which communicates using a UNIX socket to the main conntrackd daemon (the one which is actually communicating with the other node). A failback boot procedure may look like this:

  1. system boots
  2. firewall ruleset loading
  3. networking up
  4. conntrackd up (main sync daemon: using 'conntrackd -d'; UNIX socket is opened)
  5. request sync with other node (using conntrackd -n' which uses the UNIX socket opened in step 4)

In both sysvinit and systemd cases, the step 5 may be implemented using another dummy service which performs the required operations and that depends on the main service representing the step 4.

The point here is that steps 4 and 5 suffer a very bad race condition: the conntrackd daemon may open the UNIX socket (step 4) *after* the we try to use it (step 5).

As you could probably imagine, this is the worst possible scenario: as one of the nodes of the cluster is missing important flow state information the stateful firewall will drop packets and all the network monitoring alarms will start to ring... Ironically, just after a failback operation, when all of our cluster is supposed to be up an running.

To avoid the race conditions, some typical hacks could be implemented:


After a bit of research, I found that the last point could be implemented very easily using libsystemd, so the daemon inform of its internal state to systemd [4].

The solution is elegant, simple and offers more direct benefits:

So the clear winner for conntrackd is to go with systemd, using an unit service file of Type=notify. Using sysvinit is prone to the described race condition. No serious firewall cluster with this architecture would use sysvinit.

It's obvious to me that systemd is a better technology than sysvinit. The sysvinit approach has been OK for a while, and for me it was fascinating when I started developing init scripts and learning how things worked.
But now we have systemd. It's the default in Debian. It's far better.

Conclusions: yes, I will probably reintroduce sysvinit files just to avoid any flamewar.


[0] https://lists.debian.org/debian-devel/2016/08/msg00448.html
[1] https://tracker.debian.org/pkg/conntrack-tools
[2] https://lists.debian.org/debian-devel/2016/08/msg00456.html
[3] https://sources.debian.net/src/systemd/231-4/debian/systemd.NEWS/
[4] http://git.netfilter.org/conntrack-tools/tree/src/systemd.c

Dirk Eddelbuettel: RcppCNPy 0.2.5

A maintenance release of the RcppCNPy package is now on CRAN. RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers. This release just updates a few package internals such as the vignette, the DESCRIPTION and the README.md file as well as under Travis script used for continuous integration.
Changes in version 0.2.5 (2016-08-26)
  • Synchronized code with the cnpy repository
  • Updated vignette
  • Expanded DESCRIPTION
CRANberries also provides a diffstat report for the latest release. As always, feedback is welcome and the best place to start a discussion may be the GitHub issue tickets page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

26 August 2016

Norbert Preining: ESSLLI 2016 Course: Algebraic Specification and Verification with CafeOBJ

During this year s ESSLLI (European Summer School in Logic, Language and Information) I was teaching a course on Algebraic Specification and Verification with CafeOBJ. Now that the course is over I can relax a bit and enjoy the beauty of the surrounding North Tyrol.
ESSLLI-2016-logo-darkblue-small For those who couldn t attend the ESSLLI or the course, here are the course materials: Thanks to the participants for the interesting questions and participation. What follows is the original description of the course. Abstract Ensuring correctness of complex systems like computer programs or communication protocols is gaining ever increasing importance. For correctness it does not suffice to consider the finished system, but correctness already on the level of specification is necessary, that is, before the actual implementation starts. To this aim, algebraic specification and verification languages have been conceived. They are aiming at a mathematically correct description of the properties and behavior of the systems under discussion, and in addition often allow to prove (verify) correctness within the same system. This course will give an introduction to algebraic specification, its history, logic roots, and actuality with respect to current developments. Paired with the theoretical background we give an introduction to CafeOBJ, a programming language that allows both specification and verification to be carried out together. The course should enable students to understand the theoretical background of algebraic specification, as well as enable them to read and write basic specifications in CafeOBJ. Description CafeOBJ is a specification language based on three-way extensions to many-sorted equational logic: the underlying logic is order-sorted, not just many-sorted; it admits unidirectional transitions, as well as equations; it also accommodates hidden sorts, on top of ordinary, visible sorts. A subset of CafeOBJ is executable, where the operational semantics is given by a conditional order-sorted term rewriting system. The language system CafeOBJ has been under constant development at the institute of the lecturers since the late 80ies. It is closely related to other algebraic specification languages in the OBJ family, including Maude. The CafeOBJ language and the range of verification methods and tools it supports including its support for inductive theorem proving, verification of behavioral specifications, deductive invariant proof, and reachability analysis of concurrent systems has played a key role in both extending and bringing algebraic specification techniques into contact with many software engineering applications. The following topics will be discussed: To make the lectures not too heavy , we will structure each lecture into two parts: A first part providing an introduction of some theoretical concept, or framework, and a second part dealing with actual programming and implementation. Especially for the second part of each lecture students are encouraged to use their laptops to try out code and experiment.

Matthew Garrett: Priorities in security

I read this tweet a couple of weeks ago:

and it got me thinking. Security research is often derided as unnecessary stunt hacking, proving insecurity in things that are sufficiently niche or in ways that involve sufficient effort that the realistic probability of any individual being targeted is near zero. Fixing these issues is basically defending you against nation states (who (a) probably don't care, and (b) will probably just find some other way) and, uh, security researchers (who (a) probably don't care, and (b) see (a)).

Unfortunately, this may be insufficient. As basically anyone who's spent any time anywhere near the security industry will testify, many security researchers are not the nicest people. Some of them will end up as abusive partners, and they'll have both the ability and desire to keep track of their partners and ex-partners. As designers and implementers, we owe it to these people to make software as secure as we can rather than assuming that a certain level of adversary is unstoppable. "Can a state-level actor break this" may be something we can legitimately write off. "Can a security expert continue reading their ex-partner's email" shouldn't be.

comment count unavailable comments

25 August 2016

Alessio Treglia: The Breath of Time

For centuries man has hunted, he brought the animals to pasture, cultivated fields and sailed the seas without any kind of tool to measure time. Back then, the time was not measured, but only estimated with vague approximation and its pace was enough to dictate the steps of the day and the life of man. Subsequently, for many centuries, hourglasses accompanied the civilization with the slow flow of their sand grains. About hourglasses, Ernst Junger writes in Das Sanduhrbuch 1954 (no English translation): This small mountain, formed by all the moments lost that fell on each other, it could be understood as a comforting sign that the time disappears but does not fade. It grows in depth . For the philosophers of ancient Greece, the time was just a way to measure how things move in everyday life and in any case there was a clear distinction between quantitative time (Kronos) and qualitative time (Kair s). According to Parmenides, time is guise, because its existence <Read More [by Fabio Marzocca]>

Joerg Jaspert: New gnupg-agent in Debian

In case you just upgraded to the latest gnupg-agent and used gnupg-agent as your ssh-agent you may find that ssh refuses to work with a simple but not helpful sign_and_send_pubkey: signing failed: agent refused operation This seems to come from systemd starting the agent, no longer a script at the start of the X session. And so it ends up with either no or an unusable tty. A simple 
gpg-connect-agent updatestartuptty /bye
updates that and voila, ssh agent functionality is back in. Note: This assumes you have enable-ssh-support in your ~/.gnupg/gpg-agent.conf

Francois Marier: Debugging gnome-session problems on Ubuntu 14.04

After upgrading an Ubuntu 14.04 ("trusty") machine to the latest 16.04 Hardware Enablement packages, I ran into login problems. I could log into my user account and see the GNOME desktop for a split second before getting thrown back into the LightDM login manager. The solution I found was to install this missing package: 
apt install libwayland-egl1-mesa-lts-xenial

Looking for clues in the logs The first place I looked was the log file for the login manager (/var/log/lightdm/lightdm.log) where I found the following: 
DEBUG: Session pid=12743: Running command /usr/sbin/lightdm-session gnome-session --session=gnome
DEBUG: Creating shared data directory /var/lib/lightdm-data/username
DEBUG: Session pid=12743: Logging to .xsession-errors
This told me that the login manager runs the gnome-session command and gets it to create a session of type gnome. That command line is defined in /usr/share/xsessions/gnome.desktop (look for Exec=): 
[Desktop Entry]
Name=GNOME
Comment=This session logs you into GNOME
Exec=gnome-session --session=gnome
TryExec=gnome-shell
X-LightDM-DesktopName=GNOME
I couldn't see anything unexpected there, but it did point to another log file (~/.xsession-errors) which contained the following: 
Script for ibus started at run_im.
Script for auto started at run_im.
Script for default started at run_im.
init: Le processus gnome-session (GNOME) main (11946) s'est achev  avec l' tat 1
init: D connect  du bus D-Bus notifi 
init: Le processus logrotate main (11831) a  t  tu  par le signal TERM
init: Le processus update-notifier-crash (/var/crash/_usr_bin_unattended-upgrade.0.crash) main (11908) a  t  tu  par le signal TERM
Seaching for French error messages isn't as useful as searching for English ones, so I took a look at /var/log/syslog and found this: 
gnome-session[4134]: WARNING: App 'gnome-shell.desktop' exited with code 127
gnome-session[4134]: WARNING: App 'gnome-shell.desktop' exited with code 127
gnome-session[4134]: WARNING: App 'gnome-shell.desktop' respawning too quickly
gnome-session[4134]: CRITICAL: We failed, but the fail whale is dead. Sorry....
It looks like gnome-session is executing gnome-shell and that this last command is terminating prematurely. This would explain why gnome-session exits immediately after login.

Increasing the amount of logging In order to get more verbose debugging information out of gnome-session, I created a new type of session (GNOME debug) by copying the regular GNOME session: 
cp /usr/share/xsessions/gnome.desktop /usr/share/xsessions/gnome-debug.desktop
and then adding --debug to the command line inside gnome-debug.desktop: 
[Desktop Entry]
Name=GNOME debug
Comment=This session logs you into GNOME debug
Exec=gnome-session --debug --session=gnome
TryExec=gnome-shell
X-LightDM-DesktopName=GNOME debug
After restarting LightDM (service lightdm restart), I clicked the GNOME logo next to the password field and chose GNOME debug before trying to login again. This time, I had a lot more information in ~/.xsession-errors: 
gnome-session[12878]: DEBUG(+): GsmAutostartApp: starting gnome-shell.desktop: command=/usr/bin/gnome-shell startup-id=10d41f1f5c81914ec61471971137183000000128780000
gnome-session[12878]: DEBUG(+): GsmAutostartApp: started pid:13121
...
/usr/bin/gnome-shell: error while loading shared libraries: libwayland-egl.so.1: cannot open shared object file: No such file or directory
gnome-session[12878]: DEBUG(+): GsmAutostartApp: (pid:13121) done (status:127)
gnome-session[12878]: WARNING: App 'gnome-shell.desktop' exited with code 127
which suggests that gnome-shell won't start because of a missing library.

Finding the missing library To find the missing library, I used the apt-file command: 
apt-file update
apt-file search libwayland-egl.so.1
and found that this file is provided by the following packages:
  • libhybris
  • libwayland-egl1-mesa
  • libwayland-egl1-mesa-dbg
  • libwayland-egl1-mesa-lts-utopic
  • libwayland-egl1-mesa-lts-vivid
  • libwayland-egl1-mesa-lts-wily
  • libwayland-egl1-mesa-lts-xenial
Since I installed the LTS Enablement stack, the package I needed to install to fix this was libwayland-egl1-mesa-lts-xenial. I filed a bug for this on Launchpad.

Next.

Previous.