Search Results: "mez"

7 February 2017

Olivier Berger: Making Debian stable/jessie images for OpenStack with bootstrap-vz and cloud-init

I m investigating the creation of VM images for different virtualisation solutions. Among the target platforms is a destop as a service platform based on an OpenStack public cloud. We ve been working with bootstrap-vz for creating VMs for Vagrant+VirtualBox so I wanted to test its use for OpenStack. There are already pre-made images available, including official Debian ones, but I like to be able to re-create things instead of depending on some external magic (which also means to be able to optimize, customize and avoid potential MitM, of course). It appears that bootstrap-vz can be used with cloud-init provided that some bits of config are specified. In particular the cloud_init plugin of bootstrap-vz requires a metadata_source set to NoCloud, ConfigDrive, OpenStack, Ec2 . Note we explicitely spell it OpenStack and not Openstack as was mistakenly done in the default Debian cloud images (see https://bugs.debian.org/854482). The following snippet of manifest provides the necessary bits :
---
name: debian- system.release - system.architecture - %Y %m %d 
provider:
  name: kvm
  virtio_modules:
  - virtio_pci
  - virtio_blk
bootstrapper:
  workspace: /target
  # create or reuse a tarball of packages
  tarball: true
system:
  release: jessie
  architecture: amd64
  bootloader: grub
  charmap: UTF-8
  locale: en_US
  timezone: UTC
volume:
  backing: raw
  partitions:
    #type: gpt
    type: msdos
    root:
      filesystem: ext4
      size: 4GiB
    swap:
      size: 512MiB
packages:
  # change if another mirror is closer
  mirror: http://ftp.fr.debian.org/debian/
plugins:
  root_password:
    password: whatever
  cloud_init:
    username: debian
    # Note we explicitely spell it 'OpenStack' and not 'Openstack' as done in the default Debian cloud images (see https://bugs.debian.org/854482)
    metadata_sources: NoCloud, ConfigDrive, OpenStack, Ec2
  # admin_user:
  #   username: Administrator
  #   password: Whatever
  minimize_size:
    # reduce the size by around 250 Mb
    zerofree: true
I ve tested this with the bootstrap-vz version in stretch/testing (0.9.10+20170110git-1) for creating jessie/stable image, which were booted on the OVH OpenStack public cloud. YMMV. Hope this helps

5 February 2017

Bits from Debian: Debian welcomes its Outreachy interns

Outreachy logo Better late than never, we'd like to welcome our three Outreachy interns for this round, lasting from the 6th of December 2016 to the 6th of March 2017. Elizabeth Ferdman is working in the Clean Room for PGP and X.509 (PKI) Key Management. Maria Glukhova is working in Reproducible builds for Debian and free software. Urvika Gola is working in improving voice, video and chat communication with free software. From the official website: Outreachy helps people from groups underrepresented in free and open source software get involved. We provide a supportive community for beginning to contribute any time throughout the year and offer focused internship opportunities twice a year with a number of free software organizations. The Outreachy program is possible in Debian thanks to the effort of Debian developers and contributors that dedicate part of their free time to mentor students and outreach tasks, and the help of the Software Freedom Conservancy, who provides administrative support for Outreachy, as well as the continued support of Debian's donors, who provide funding for the internships. Debian will also participate in the next round for Outreachy, during the summer of 2017. More details will follow in the next weeks. Join us and help extend Debian! You can follow the work of the Outreachy interns reading their blogs (they are syndicated in Planet Debian), and chat with us in the #debian-outreach IRC channel and mailing list. Congratulations, Elizabeth, Maria and Urvika!

16 January 2017

Maria Glukhova: APK, images and other stuff.

2 more weeks of my awesome Outreachy journey have passed, so it is time to make an update on my progress. I continued my work on improving diffoscope by fixing bugs and completing wishlist items. These include:

Improving APK support I worked on #850501 and #850502 to improve the way diffoscope handles APK files. Thanks to Emanuel Bronshtein for providing clear description on how to reproduce these bugs and ideas on how to fix them. And special thanks to Chris Lamb for insisting on providing tests for these changes! That part actually proved to be little more tricky, and I managed to mess up with these tests (extra thanks to Chris for cleaning up the mess I created). Hope that also means I learned something from my mistakes. Also, I was pleased to see F-droid Verification Server as a sign of F-droid progress on reproducible builds effort - I hope these changes to diffoscope will help them!

Adding support for image metadata That came from #849395 - a request was made to compare image metadata along with image content. Diffoscope has support for three types of images: JPEG, MS Windows Icon (*.ico) and PNG. Among these, PNG already had good image metadata support thanks to sng tool, so I worked on .jpeg and .ico files support. I initially tried to use exiftool for extracting metadata, but then I discovered it does not handle .ico files, so I decided to use a bigger force - ImageMagick s identify - for this task. I was glad to see it had that handy -format option I could use to select only the necessary fields (I found their -verbose, well, too verbose for the task) and presenting them in the defined form, negating the need of filtering its output. What was particulary interesting and important for me in terms of learning: while working on this feature, I discovered that, at the moment, diffoscope could not handle .ico files at all - img2txt tool, that was used for retrieving image content, did not support that type of images. But instead of recognizing this as a bug and resolving it, I started to think of possible workaround, allowing for retrieving image metadata even after retrieving image content failed. Definetely not very good thinking. Thanks Mattia Rizzolo for actually recognizing this as a bug and filing it, and Chris Lamb for fixing it!

Other work

Order-like differences, part 2 In the previous post, I mentioned Lunar s suggestion to use hashing for finding order-like difference in wide variety of input data. I implemented that idea, but after discussion with my mentor, we decided it is probably not worth it - this change would alter quite a lot of things in core modules of diffoscope, and the gain would be not really significant. Still, implementing that was an important experience for me, as I had to hack on deepest and, arguably, most difficult modules of diffoscope and gained some insight on how they work.

Comparing with several tools (work in progress) Although my initial motivation for this idea was flawed (the workaround I mentioned earlier for .ico files), it still might be useful to have a mechanism that would allow to run several commands for finding difference, and then give the output of those that succeed, failing if and only if they all have failed. One possible case when it might happen is when we use commands coming from different tools, and one of them is not installed. It would be nice if we still used the other and not the uninformative binary diff (that is a default fallback option for when something goes wrong with more clever comparison). I am still in process of polishing this change, though, and still in doubt if it is needed at all.

Side note - Outreachy and my university progress In my Outreachy application, I promised that if I am selected into this round, I will do everything I can to unload the required time period from my university time commitements. I did that by moving most of my courses to the first half of the academic year. Now, the main thing that is left for me to do is my Master s thesis. I consulted my scientific advisors from both universities that I am formally attending (SFEDU and LUT - I am in double degree program), and as a result, they agreed to change my Master s thesis topic to match my Outreachy work. Now, that should have sounded like an excellent news - merging these activities together actually mean I can allocate much more time to my work on reproducible builds, even beyond the actual internship time period. That was intended to remove a burden from my shoulders. Still, I feel a bit uneasy. The drawback of this decision lies in fact I have no idea on how to write scientific report based on pure practical work. I know other students from my universities have done such things before, but choosing my own topic means my scientific advisors can t help me much - this is just out of their area of expertise. Well, wish me luck - I m up to the challenge!

11 January 2017

Reproducible builds folks: Reproducible Builds: week 89 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday January 1 and Saturday January 7 2017: GSoC and Outreachy updates Toolchain development Packages reviewed and fixed, and bugs filed Chris Lamb: Dhole: Reviews of unreproducible packages 13 package reviews have been added, 4 have been updated and 6 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been added/updated: Upstreaming of reproducibility fixes Merged: Opened: Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by: diffoscope development diffoscope 67 was uploaded to unstable by Chris Lamb. It included contributions from :
[ Chris Lamb ]
* Optimisations:
  - Avoid multiple iterations over archive by unpacking once for an ~8X
    runtime optimisation.
  - Avoid unnecessary splitting and interpolating for a ~20X optimisation
    when writing --text output.
  - Avoid expensive diff regex parsing until we need it, speeding up diff
    parsing by 2X.
  - Alias expensive Config() in diff parsing lookup for a 10% optimisation.
* Progress bar:
  - Show filenames, ELF sections, etc. in progress bar.
  - Emit JSON on the the status file descriptor output instead of a custom
    format.
* Logging:
  - Use more-Pythonic logging functions and output based on __name__, etc.
  - Use Debian-style "I:", "D:" log level format modifier.
  - Only print milliseconds in output, not microseconds.
  - Print version in debug output so that saved debug outputs can standalone
    as bug reports.
* Profiling:
  - Also report the total number of method calls, not just the total time.
  - Report on the total wall clock taken to execute diffoscope, including
    cleanup.
* Tidying:
  - Rename "NonExisting" -> "Missing".
  - Entirely rework diffoscope.comparators module, splitting as many separate
    concerns into a different utility package, tidying imports, etc.
  - Split diffoscope.difference into diffoscope.diff, etc.
  - Update file references in debian/copyright post module reorganisation.
  - Many other cleanups, etc.
* Misc:
  - Clarify comment regarding why we call python3(1) directly. Thanks to J r my
    Bobbio <lunar@debian.org>.
  - Raise a clearer error if trying to use --html-dir on a file.
  - Fix --output-empty when files are identical and no outputs specified.
[ Reiner Herrmann ]
* Extend .apk recognition regex to also match zip archives (Closes: #849638)
[ Mattia Rizzolo ]
* Follow the rename of the Debian package "python-jsbeautifier" to
  "jsbeautifier".
[ siamezzze ]
* Fixed no newline being classified as order-like difference.
reprotest development reprotest 0.5 was uploaded to unstable by Chris Lamb. It included contributions from:
[ Ximin Luo ]
* Stop advertising variations that we're not actually varying.
  That is: domain_host, shell, user_group.
* Fix auto-presets in the case of a file in the current directory.
* Allow disabling build-path variations. (Closes: #833284)
* Add a faketime variation, with NO_FAKE_STAT=1 to avoid messing with
  various buildsystems. This is on by default; if it causes your builds
  to mess up please do file a bug report.
* Add a --store-dir option to save artifacts.
Other contributions (not yet uploaded): reproducible-builds.org website development tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen and Vagrant Cascadian, reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

9 January 2017

Dirk Eddelbuettel: RcppCCTZ 0.2.0

A new version, now at 0.2.0, of RcppCCTZ is now on CRAN. And it brings a significant change: windows builds! Thanks to Dan Dillon who dug deep enough into the libc++ sources from LLVM to port the std::get_time() function that is missing from the 4.* series of g++. And with Rtools being fixed at g++-4.9.3 this was missing for us here. Now we can parse dates for use by RcppCCTZ on Windows as well. That is important not only for RcppCCTZ but also particularly for the one package (so far) depending on it: nanotime. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo -- and for which R on Windows ships its own copy we can use. RcppCCTZ connects this library to R by relying on Rcpp. The RcppCCTZ page has a few usage examples, as does the post announcing the previous release. The changes in this version are summarized here:

Changes in version 0.2.0 (2017-01-08)
  • Windows compilation was enabled by defining OFFSET() and ABBR() for MinGW (#10 partially addressing #9)
  • Windows use completed with backport of std::get_time from LLVM's libc++ to enable strptime semantics (Dan Dillon in #11 completing #9)
  • Timezone information on Windows is supplied via R's own copy of zoneinfo with TZDIR set (also #10)
  • The interface to formatDouble was cleaned up

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

3 January 2017

Reproducible builds folks: Reproducible Builds: week 88 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday December 25 and Saturday December 31 2016: Media coverage Reproducible bugs filed Chris West: Chris Lamb: Rob Browning: Reviews of unreproducible packages 7 package reviews have been added, 12 have been updated and 14 have been removed in this week, adding to our knowledge about identified issues. 2 issue types have been updated: Weekly QA work During our reproducibility testing, the following FTBFS bugs have been detected and reported by: diffoscope development strip-nondeterminism development try.diffoscope.org development tests.reproducible-builds.org Misc. This week's edition was written by Chris Lamb, Holger Levsen and was reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

Maria Glukhova: Getting to know diffoscope better

I apologize to all potential readers of this blog for not writing a comprehensive Introduction post with details of the project I am taking part in during my internship, as well as some story about how I ended up there. Let me just say that I was a Debian user for years when I discovered it is taking part in Outreachy as one of organisations. Their Reproducible Builds effort has a noble goal and a bunch of great people behind it - I had no chances not to get excited by it. Looking for a place where my skills could be of any use, I discovered diffoscope - the tool for in-depth comparassion of files, archives etc. My mentor, Mattia Rizzolo, supported my decision to work on it, so now I am concentrating my efforts on improving diffoscope. As my first steps, I am doing small (but hopefully still somewhat important) job of fixing existing bugs. It helps me to better understand how diffoscope works, as well as introduces me to the workflow of opensource development. During December, I have done several small contributions, mostly fixing bugs.

Test data and jessie-backports First of them could be somewhat called cleaning up after my own mistake, although that mistake wasn t trivial. During the application period, I have fixed a bug with diffoscope failing while comparing symlinks to directory. That was a small change, but I included some tests for that case anyway. And that actually caused problems. With these tests, I included test data: two folders with symlinks. All was good in unstable version of Debian, but in jessie-backports, that commit caused build to fail. After some digging, I discovered the problem was caused by build process including copying that data. That was done using shutils Python module, and older version of that module, included in jessie, could not handle copying symlinks to directory properly. Thanks to my mentor for giving me a hint on how to resolve this: using temporary folders and creating these symlinks at runtime. That way, we ensured tests run without problems during build process on jessie. What have I learned: A great deal, actually. I spent too much time on that one, but I learned how to build packages, what happens during dpkg-buildpackage run and what debhelper tools are for. I also learned a bit about what chroot is and how to use it for testing.

ICC profile files and file type recognizing regexp Another one was also about failing tests and, therefore, failing build. Failing tests were all due to ICC files were not recognized by diffoscope. Turned out libmagic got an update which changed the description of ICC profile files. Diffoscope was relying on regexp applied to file type description to recognize the file, so I changed regexp to reflect the changes in libmagic. What have I learned: How diffoscope recognizes file types. Got me thinking: maybe there is a better way? That regexp-based approach is doomed to cause problems with every file type description change. I have this question still lingering in my mind - maybe I will come up with an idea later.

Order-like difference in text files Next, I decided to do something a bit bigger and fullfilled a feature request. That request was for detecting order-like difference in text files (when files has the same lines, but in different order). I did it by collecting added and removed lines in diff output in lists, sorting and then comparing them. Sadly, I forgot about one particular case - when one of the files is missing the newline at the end of file. I was kindly reminded of that quite soon in comments on the bug-tracker (thanks danielsh!) and have already fixed that. I also recieved feedback on how better implement it deeper in the diffoscope - not using the results of diff, but rather comparing sum of hashes of the lines directly in the difference module. I am yet to try that. What have I learned: That a call to diff is actually the slowest part of the diffoscope run when done on two big text files. Could it help somehow in speeding it up? I don t know yet. I also learned to comment on bugs in Debian bugtracker and was surprised by how much feedback I got. Thanks to my mentor for pushing me to do that - I definetely need to overcome my fear of communications to be more effective!

Random FTBFS There was also a very nasty bug that caused diffoscope to fail to be built from source randomly, failing with non-informative Fatal Python error: deallocated None. It already seemed strange when it was first reported; It got only more strange when suddenly that bug ceased to be reproducible. We hoped that would mean that bug was caused by some external tool, and was fixed there. Turns out it was not that easy. I tested this on two separate computers and on virtual machine; I used different versions of diffoscope. Well. Seems like that bug is still somehow tied to diffoscope version and not some external tool version - I still can do git checkout 64 and be able to reproduce the bug (still randomly, though). Although I spent quite a lot of time on that one, the only result was the information about connection between bug apperances and diffoscope version. I still wasn t able to get to the root of the problem - hopefully, someone else will be able to, given the information I found. What have I learned: git-bisect! Thanks to my friend for pointing me to it, that tool came handy in that situation. Also, got some experience in catching nasty bugs like that (pity that no experience in squashing them). I had some extra time commitements in December, one of them (Reproducible Builds Summit II) connected to my internship and one (my exam session in university) not. In January, I should be able to allocate more time to that work - I hope it will help me achieve more significant results. Many thanks to Mattia Rizzolo, Chris Lamb, Holger Levsen and all other folks of Reproducible Builds project - I cannot stress enough how important your support is to me. Wish you all a great 2017!

24 December 2016

Dirk Eddelbuettel: anytime 0.2.0: Feature, fixes and tests!

A brand new anytime package just arrived at CRAN. This is release number eight, evenly spread with over two per month, since the initial release in September. Needless to say I have been told off not to make this many releases. As they say, no good deed goes unpunished. anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, ... format to either POSIXct or Date objects -- and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples. This releases does a few things: The following is a quick illustration
R> library(anytime)
R> p <- anytime("2010-01-02 03:04:05.123456")
R> p
[1] "2010-01-02 03:04:05.123456 CST"
R> iso8601(p)
[1] "2010-01-02 03:04:05"
R> rfc2822(p)
[1] "Sat, 02 Jan 2010 03:04:05.123456 -0600"
R> rfc3339(p)
[1] "2010-01-02T03:04:05.123456-0600"
R> 
For symmetry, it also works for dates, but is less detailed
R> jl <- anydate("July 04, 1789")
R> jl
[1] "1789-07-04"
R> iso8601(jl)
[1] "1789-07-04"
R> rfc2822(jl)
[1] "Sat, 04 Jul 1789"
R> rfc3339(jl)
[1] "1789-07-04"
R> 
Of course, all commands are also fully vectorised. See the anytime page, or the GitHub README.md for more examples.

Changes in anytime version 0.2.0 (2016-12-24)
  • Added (exported) helper functions iso8601(), rfc2822() and rfc3339() to format date(time) objects according to standards
  • Conversion to dates is now more robust thanks to improved internal processing (PR #39 closing #36)
  • The ISO 8601 format is now recognised, however the timezone information is not parsed by Boost Date_Time (which is a known upstream limitation) (PR #38 closing #37)
  • The 'allFormats.R' test script was significantly strengthened (#40)
  • Test scripts like 'simpleTests.R' have as also been strengthened (#41); on Windows and in one file two tests need to be skipped.
  • A new 'bulkTest.R' test script was added testing parsing against what R returns

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

18 December 2016

Jonathan McDowell: Timezones + static blog generation

So, it turns out when you move to static blog generation and do the generation on your laptop, which is usually in the timezone you re currently physically located, it can cause URLs to change. Especially if you re prone to blogging late at night, which can result in even just a shift to DST changing things. I ve forced jekyll to UTC by adding timezone: 'UTC' to the config, and ensuring all the posts now have timezones for when they were written (a lot of the imported ones didn t), so hopefully things should be stable from here on.

12 December 2016

Dirk Eddelbuettel: RcppCCTZ 0.1.0

A new version 0.1.0 of RcppCCTZ arrived on CRAN this morning. It brings a number of new or updated things, starting with new upstream code from CCTZ as well as a few new utility functions. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. It requires only a proper C++11 compiler and the standard IANA time zone data base which standard Unix, Linux, OS X, ... computers tend to have in /usr/share/zoneinfo. RcppCCTZ connects this library to R by relying on Rcpp. A nice example is the helloMoon() function (based on an introductory example in the CCTZ documentation) showing the time when Neil Armstrong took a small step, relative to local time in New York and Sydney:
R> library(RcppCCTZ)
R> helloMoon(verbose=TRUE)
1969-07-20 22:56:00 -0400
1969-07-21 12:56:00 +1000
                   New_York                      Sydney 
"1969-07-20 22:56:00 -0400" "1969-07-21 12:56:00 +1000" 
R> 
The new formating and parsing functions are illustrated below with default arguments for format strings and timezones. All this can be customized as usual.
R> example(formatDatetime)
frmtDtR> now <- Sys.time()
frmtDtR> formatDatetime(now)            # current (UTC) time, in full precision RFC3339
[1] "2016-12-12T13:21:03.866711+00:00"
frmtDtR> formatDatetime(now, tgttzstr="America/New_York")  # same but in NY
[1] "2016-12-12T08:21:03.866711-05:00"
frmtDtR> formatDatetime(now + 0:4)     # vectorised
[1] "2016-12-12T13:21:03.866711+00:00" "2016-12-12T13:21:04.866711+00:00" "2016-12-12T13:21:05.866711+00:00"
[4] "2016-12-12T13:21:06.866711+00:00" "2016-12-12T13:21:07.866711+00:00"
R> example(parseDatetime)
prsDttR> ds <- getOption("digits.secs")
prsDttR> options(digits.secs=6) # max value
prsDttR> parseDatetime("2016-12-07 10:11:12",        "%Y-%m-%d %H:%M:%S");   # full seconds
[1] "2016-12-07 04:11:12 CST"
prsDttR> parseDatetime("2016-12-07 10:11:12.123456", "%Y-%m-%d %H:%M:%E*S"); # fractional seconds
[1] "2016-12-07 04:11:12.123456 CST"
prsDttR> parseDatetime("2016-12-07T10:11:12.123456-00:00")  ## default RFC3339 format
[1] "2016-12-07 04:11:12.123456 CST"
prsDttR> now <- trunc(Sys.time())
prsDttR> parseDatetime(formatDatetime(now + 0:4))               # vectorised
[1] "2016-12-12 07:21:17 CST" "2016-12-12 07:21:18 CST" "2016-12-12 07:21:19 CST"
[4] "2016-12-12 07:21:20 CST" "2016-12-12 07:21:21 CST"
prsDttR> options(digits.secs=ds)
R>
Changes in this version are summarized here:

Changes in version 0.1.0 (2016-12-11)
  • Synchronized with CCTZ upstream.
  • New parsing and formating helpers for Datetime vectors
  • New parsing and formating helpers for (two) double vectors representing full std::chrono nanosecond resolutions
  • Updated documentation and examples.

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

17 November 2016

Reproducible builds folks: Reproducible Builds: week 81 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday November 6 and Saturday November 12 2016: Media coverage Matthew Garrett blogged about Tor, TPMs and service integrity attestation and how reproducible builds are the base for systems integrity. The Linux Foundation announced renewed funding for us as part of the Core Infrastructure Initiative. Thank you! Outreachy updates Maria Glukhova has been accepted into the Outreachy winter internship and will work with us the Debian reproducible builds team. To quote her words
siamezzze: I've been accepted to #outreachy winter internship - going to
work with Debian reproducible builds team. So excited about that! <3
Debian
Toolchain development and fixes dpkg: debrebuild: Bugs filed Chris Lamb: Daniel Shahaf: Niko Tyni: Reiner Herrman: Reviews of unreproducible packages 136 package reviews have been added, 5 have been updated and 7 have been removed in this week, adding to our knowledge about identified issues. 3 issue types have been updated: Weekly QA work During of reproducibility testing, some FTBFS bugs have been detected and reported by: diffoscope development A new version of diffoscope 62~bpo8+1 was uploaded to jessie-backports by Mattia Rizzolo. Meanwhile in git, Ximin Luo greatly improved speed by fixing a O(n2) lookup which was causing diffs of large packages such as GCC and glibc to take many more hours than was necessary. When this commit is released, we should hopefully see full diffs for such packages again. Currently we have 197 source packages which - when built - diffoscope fails to analyse. buildinfo.debian.net development tests.reproducible-builds.org Debian: reproducible-builds.org website F-Droid was finally added to our list of partner projects. (This was an oversight and they had already been working with us for some time.) Misc. This week's edition was written by Ximin Luo and Holger Levsen and reviewed by a bunch of Reproducible Builds folks on IRC.

8 November 2016

Dirk Eddelbuettel: gettz 0.0.3

A minor release 0.0.3 of gettz arrived on CRAN two days ago. gettz provides a possible fallback in situations where Sys.timezone() fails to determine the system timezone. That can happen when e.g. the file /etc/localtime somehow is not a link into the corresponding file with zoneinfo data in, say, /usr/share/zoneinfo. This release adds a second #ifdef to permit builds on Windows for the previous R release (ie r-oldrel-windows). No new code, or new features. Courtesy of CRANberries, there is a comparison to the previous release. More information is on the gettz page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

31 October 2016

James Bromberger: The Debian Cloud Sprint 2016

I m at an airport, about to board the first of three flights across the world, from timezone +8 to timezone -8. I ll be in transit 27 hours to get to Seattle, Washington state. I m leaving my wife and two young children behind. My work has given me a days worth of leave under the Corporate Social Responsibility program, and I m taking three days annual leave, to do this. 27 hours each way in transit, for 3 days on the ground. Why? Backstory I started playing in technology as a kid in the 1980s; my first PC was a clone (as they were called) 286 running MS-DOS. It was clunky, and the most I could do to extend it was to write batch scripts. As a child I had no funds for commercial compilers, no network connections (this was pre Internet in Australia), no access to documentation, and no idea where to start programming properly. It was a closed world. I hit university in the summer of 1994 to study Computer Science and French. I d heard of Linux, and soon found myself installing the Linux distributions of the day. The Freedom of the licensing, the encouragement to use, modify, share, was in stark contrast to the world of consumer PCs of the late 1980 s. It was there at the UCC at UWA I discovered Debian. Some of the kind network/system admins at the University maintained a Debian mirror on the campus LAN, updated regularly and always online. It was fast, and more importantly, free for me to access. Back in the 1990s, bandwidth in Australia was incredibly expensive. The vast distances of the country mean that bandwidth was scarce. Telcos were in races to put fiber between Perth and the Eastern States, and without that in place, IP connectivity was constrained, and thus costly. Over many long days and nights I huddled down, learning window managers, protocols, programming and scripting languages. I became a system/network administrator, web developer, dev ops engineer, etc. My official degree workload, algorithmic complexity, protocol stacks, were interesting, but fiddling with Linux based implementations was practical. Volunteer After years of consuming the output of Debian and running many services with it I decided to put my hand up and volunteer as a Debian Developer: it was time to give back. I had benefited from Debian, and I saw others benefit from it as well. As the 2000 s started, I had my PGP key in the Debian key ring. I had adopted a package and was maintaining it load balancing Apache web servers. The web was yet to expand to the traffic levels you see today; most web sites were served from one physical web server. Site Reliability Engineering was a term not yet dreamed of. What became more apparent was the applicability of Linux, Open Source, and in my line-of-sight Debian to a wider community beyond myself and my university peers. Debain was being used to revive recycled computers that were being donated to charities; in some cases, unable to transfer commercial software licenses with the hardware that was no longer required by organisations that had upgraded. It appeared that Debian was being used as a baseline above which society in general had access to fundamental capability of computing and network services. The removal of subscriptions, registrations, and the encouragement of distribution meant this occurred at rates that could never be tracked, and more importantly, the consensus was that it should not be automatically tracked. The privacy of the user is paramount more important than some statistics for the Developer to ponder. When the Bosnia-Herzegovina war ended in 1995, I recall an email from academics there, having found some connectivity, writing to ask if they would be able to use Debian as part of their re-deployment of services for the Tertiary institutions in the region. This was an unnecessary request as Debian GNU/Linux is freely available, but it was a reminder that, for the country to have tried to procure commercial solutions at that time would have been difficult. Instead, those that could do the task just got on with it. There s been many similar project where the grass-roots organisations non profits, NGOs, and even just loose collectives of individuals have turned to Linux, Open Source, and sometimes Debian to solve their problems. Many fine projects have been established to make technology accessible to all, regardless of race, gender, nationality, class, or any other label society has used to divide humans. Big hat tip to Humanitarian Open Street Map, Serval Project. I ve always loved Debian s position on being the Universal operating system. Its vast range of packages and wide range of computing architectures supported means that quite often a litmus test of is project X a good project? was met with is it packaged for Debian? . That wide range of architectures has meant that administrators of systems had fewer surprises and a faster adoption cycle when changing platforms, such as the switch from x86 32 bit to x86 64 bit. Enter the Cloud I first laid eyes on the AWS Cloud in 2008. It was nothing like the rich environment you see today. The first thing I looked for was my favourite operating system, so that what I already knew and was familiar with was available in this environment to minimise the learning curve. However there were no official images, which was disconcerting. In 2012 I joined AWS as an employee. Living in Australia they hired me into the field sales team as a Solution Architect a sort of pre-sales tech with a customer focused depth in security. It was a wonderful opportunity, and I learnt a great deal. It also made sense (to me, at least) to do something about getting Debian s images blessed. It turned out, that I had to almost define what that was: images endorsed by a Debian Developer, handed to the AWS Marketplace team. And so since 2013 I have done so, keeping track of Debian s releases across the AWS regions, collaborating with other Debian folk on other cloud platforms to attempt a unified approach to generating and maintaining these images. This included (for a stint) generating them into the AWS GovCloud Region, and still into the AWS China (Beijing) Region the other side of the so-called Great Firewall of China. So why the trip? We ve had focus groups at the Debconf (Debian conference) around the world, but its often difficult to get the right group of people in the same rooms at the same time. So the proposal was to hold a focused Debian Cloud Sprint. Google was good enough to host this, for all the volunteers across all the cloud providers. Furthermore, donated funds were found to secure the travel for a set of people to attend who otherwise could not. I was lucky enough to be given a flight. So here I am, in the terminal in Australia: my kids are tucked up in bed, dreaming of the candy they just collected for Halloween. It will be a draining week I am sure, but if it helps set and improve the state of Debian then its worth it.

21 October 2016

Dirk Eddelbuettel: anytime 0.0.4: New features and fixes

A brand-new release of anytime is now on CRAN following the three earlier releases since mid-September. anytime aims to convert anything in integer, numeric, character, factor, ordered, ... format to POSIXct (or Date) objects -- and does so without requiring a format string. See the anytime page for a few examples. With release 0.0.4, we add two nice new features. First, NA, NaN and Inf are now simply skipped (similar to what the corresponding Base R functions do). Second, we now also accept large numeric values so that, _e.g., anytime(as.numeric(Sys.time()) also works, effectively adding another input type. We also have squashed an issue reported by the 'undefined behaviour' sanitizer, and the widened the test for when we try to deploy the gettz package get missing timezone information. A quick example of the new features:
anydate(c(NA, NaN, Inf, as.numeric(as.POSIXct("2016-09-01 10:11:12"))))
[1] NA           NA           NA           "2016-09-01"
The NEWS file summarises the release:

Changes in anytime version 0.0.4 (2016-10-20)
  • Before converting via lexical_cast, assign to atomic type via template logic to avoid an UBSAN issue (PR #15 closing issue #14)
  • More robust initialization and timezone information gathering.
  • More robust processing of non-finite input also coping with non-finite values such as NA, NaN and Inf which all return NA
  • Allow numeric POSIXt representation on input, also creating proper POSIXct (or, if requested, Date)

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

20 October 2016

H ctor Or n Mart nez: Build a Debian package against Debian 8.0 using Download On Demand (DoD) service

In the previous post Open Build Service software architecture has been overviewed. In the current blog post, a tutorial on setting up a package build with OBS from Debian packages is presented. Steps: Generate a test environment by creating Stretch/SID VM Really, use whatever suits you best, but please create an untrusted test environment for this one. In the current tutorial it assumes $hostname is stretch , which should be stretch or sid suite. Be aware that copy & paste configuration files from current post might lead you into broken characters (i.e. ). Debian Stretch weekly netinst CD Enable experimental repository
# echo "deb http://httpredir.debian.org/debian experimental main" >> /etc/apt/sources.list.d/experimental.list
# apt-get update
Install and setup OBS server, api, worker and osc CLI packages
# apt-get install obs-server obs-api obs-worker osc
In the install process mysql database is needed, therefore if mysql server is not setup, a password needs to be provided.
When OBS API database obs-api is created, we need to pick a password for it, provide opensuse . The obs-api package will configure apache2 https webserver (creating a dummy certificate for stretch ) to serve OBS webui.
Add stretch and obs aliases to localhost entry in your /etc/hosts file.
Enable worker by setting ENABLED=1 in /etc/default/obsworker
Try to connect to the web UI https://stretch/
Login into OBS webui, default login credentials: Admin/opensuse).
From command line tool, try to list projects in OBS
 $ osc -A https://stretch ls
Accept dummy certificate and provide credentials (defaults: Admin/opensuse)
If the install proceeds as expected follow to the next step. Ensure all OBS services are running
# backend services
obsrun     813  0.0  0.9 104960 20448 ?        Ss   08:33   0:03 /usr/bin/perl -w /usr/lib/obs/server/bs_dodup
obsrun     815  0.0  1.5 157512 31940 ?        Ss   08:33   0:07 /usr/bin/perl -w /usr/lib/obs/server/bs_repserver
obsrun    1295  0.0  1.6 157644 32960 ?        S    08:34   0:07  \_ /usr/bin/perl -w /usr/lib/obs/server/bs_repserver
obsrun     816  0.0  1.8 167972 38600 ?        Ss   08:33   0:08 /usr/bin/perl -w /usr/lib/obs/server/bs_srcserver
obsrun    1296  0.0  1.8 168100 38864 ?        S    08:34   0:09  \_ /usr/bin/perl -w /usr/lib/obs/server/bs_srcserver
memcache   817  0.0  0.6 346964 12872 ?        Ssl  08:33   0:11 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1
obsrun     818  0.1  0.5  78548 11884 ?        Ss   08:33   0:41 /usr/bin/perl -w /usr/lib/obs/server/bs_dispatch
obsserv+   819  0.0  0.3  77516  7196 ?        Ss   08:33   0:05 /usr/bin/perl -w /usr/lib/obs/server/bs_service
mysql      851  0.0  0.0   4284  1324 ?        Ss   08:33   0:00 /bin/sh /usr/bin/mysqld_safe
mysql     1239  0.2  6.3 1010744 130104 ?      Sl   08:33   1:31  \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306
# web services
root      1452  0.0  0.1 110020  3968 ?        Ss   08:34   0:01 /usr/sbin/apache2 -k start
root      1454  0.0  0.1 435992  3496 ?        Ssl  08:34   0:00  \_ Passenger watchdog
root      1460  0.3  0.2 651044  5188 ?        Sl   08:34   1:46      \_ Passenger core
nobody    1465  0.0  0.1 444572  3312 ?        Sl   08:34   0:00      \_ Passenger ust-router
www-data  1476  0.0  0.1 855892  2608 ?        Sl   08:34   0:09  \_ /usr/sbin/apache2 -k start
www-data  1477  0.0  0.1 856068  2880 ?        Sl   08:34   0:09  \_ /usr/sbin/apache2 -k start
www-data  1761  0.0  4.9 426868 102040 ?       Sl   08:34   0:29 delayed_job.0
www-data  1767  0.0  4.8 425624 99888 ?        Sl   08:34   0:30 delayed_job.1
www-data  1775  0.0  4.9 426516 101708 ?       Sl   08:34   0:28 delayed_job.2
nobody    1788  0.0  5.7 496092 117480 ?       Sl   08:34   0:03 Passenger RubyApp: /usr/share/obs/api
nobody    1796  0.0  4.9 488888 102176 ?       Sl   08:34   0:00 Passenger RubyApp: /usr/share/obs/api
www-data  1814  0.0  4.5 282576 92376 ?        Sl   08:34   0:22 delayed_job.1000
www-data  1829  0.0  4.4 282684 92228 ?        Sl   08:34   0:22 delayed_job.1010
www-data  1841  0.0  4.5 282932 92536 ?        Sl   08:34   0:22 delayed_job.1020
www-data  1855  0.0  4.9 427988 101492 ?       Sl   08:34   0:29 delayed_job.1030
www-data  1865  0.2  5.0 492500 102964 ?       Sl   08:34   1:09 clockworkd.clock
www-data  1899  0.0  0.0  87100  1400 ?        S    08:34   0:00 /usr/bin/searchd --pidfile --config /usr/share/obs/api/config/production.sphinx.conf
www-data  1900  0.1  0.4 161620  8276 ?        Sl   08:34   0:51  \_ /usr/bin/searchd --pidfile --config /usr/share/obs/api/config/production.sphinx.conf
# OBS worker
root      1604  0.0  0.0  28116  1492 ?        Ss   08:34   0:00 SCREEN -m -d -c /srv/obs/run/worker/boot/screenrc
root      1605  0.0  0.9  75424 18764 pts/0    Ss+  08:34   0:06  \_ /usr/bin/perl -w ./bs_worker --hardstatus --root /srv/obs/worker/root_1 --statedir /srv/obs/run/worker/1 --id stretch:1 --reposerver http://obs:5252 --jobs 1
Create an OBS project for Download on Demand (DoD) Create a meta project file:
$ osc -A https://stretch:443 meta prj Debian:8 -e
<project name= Debian:8 >
<title>Debian 8 DoD</title>
<description>Debian 8 DoD</description>
<person userid= Admin role= maintainer />
<repository name= main >
<download arch= x86_64 url= http://deb.debian.org/debian/jessie/main repotype= deb />
<arch>x86_64</arch>
</repository>
</project>
Visit webUI to check project configuration Create a meta project configuration file:
$ osc -A https://stretch:443 meta prjconf Debian:8 -e
Add the following file, as found at build.opensuse.org
Repotype: debian
# create initial user
Preinstall: base-passwd
Preinstall: user-setup
# required for preinstall images
Preinstall: perl
# preinstall essentials + dependencies
Preinstall: base-files base-passwd bash bsdutils coreutils dash debconf
Preinstall: debianutils diffutils dpkg e2fslibs e2fsprogs findutils gawk
Preinstall: gcc-4.9-base grep gzip hostname initscripts insserv libacl1
Preinstall: libattr1 libblkid1 libbz2-1.0 libc-bin libc6 libcomerr2 libdb5.3
Preinstall: libgcc1 liblzma5 libmount1 libncurses5 libpam-modules
Preinstall: libpcre3 libsmartcols1
Preinstall: libpam-modules-bin libpam-runtime libpam0g libreadline6
Preinstall: libselinux1 libsemanage-common libsemanage1 libsepol1 libsigsegv2
Preinstall: libslang2 libss2 libtinfo5 libustr-1.0-1 libuuid1 login lsb-base
Preinstall: mount multiarch-support ncurses-base ncurses-bin passwd perl-base
Preinstall: readline-common sed sensible-utils sysv-rc sysvinit sysvinit-utils
Preinstall: tar tzdata util-linux zlib1g
Runscripts: base-passwd user-setup base-files gawk
VMinstall: libdevmapper1.02.1
Order: user-setup:base-files
# Essential packages (this should also pull the dependencies)
Support: base-files base-passwd bash bsdutils coreutils dash debianutils
Support: diffutils dpkg e2fsprogs findutils grep gzip hostname libc-bin 
Support: login mount ncurses-base ncurses-bin perl-base sed sysvinit 
Support: sysvinit-utils tar util-linux
# Build-essentials
Required: build-essential
Prefer: build-essential:make
# build script needs fakeroot
Support: fakeroot
# lintian support would be nice, but breaks too much atm
#Support: lintian
# helper tools in the chroot
Support: less kmod net-tools procps psmisc strace vim
# everything below same as for Debian:6.0 (apart from the version macros ofc)
# circular dependendencies in openjdk stack
Order: openjdk-6-jre-lib:openjdk-6-jre-headless
Order: openjdk-6-jre-headless:ca-certificates-java
Keep: binutils cpp cracklib file findutils gawk gcc gcc-ada gcc-c++
Keep: gzip libada libstdc++ libunwind
Keep: libunwind-devel libzio make mktemp pam-devel pam-modules
Keep: patch perl rcs timezone
Prefer: cvs libesd0 libfam0 libfam-dev expect
Prefer: gawk locales default-jdk
Prefer: xorg-x11-libs libpng fam mozilla mozilla-nss xorg-x11-Mesa
Prefer: unixODBC libsoup glitz java-1_4_2-sun gnome-panel
Prefer: desktop-data-SuSE gnome2-SuSE mono-nunit gecko-sharp2
Prefer: apache2-prefork openmotif-libs ghostscript-mini gtk-sharp
Prefer: glib-sharp libzypp-zmd-backend mDNSResponder
Prefer: -libgcc-mainline -libstdc++-mainline -gcc-mainline-c++
Prefer: -libgcj-mainline -viewperf -compat -compat-openssl097g
Prefer: -zmd -OpenOffice_org -pam-laus -libgcc-tree-ssa -busybox-links
Prefer: -crossover-office -libgnutls11-dev
# alternative pkg-config implementation
Prefer: -pkgconf
Prefer: -openrc
Prefer: -file-rc
Conflict: ghostscript-library:ghostscript-mini
Ignore: sysvinit:initscripts
Ignore: aaa_base:aaa_skel,suse-release,logrotate,ash,mingetty,distribution-release
Ignore: gettext-devel:libgcj,libstdc++-devel
Ignore: pwdutils:openslp
Ignore: pam-modules:resmgr
Ignore: rpm:suse-build-key,build-key
Ignore: bind-utils:bind-libs
Ignore: alsa:dialog,pciutils
Ignore: portmap:syslogd
Ignore: fontconfig:freetype2
Ignore: fontconfig-devel:freetype2-devel
Ignore: xorg-x11-libs:freetype2
Ignore: xorg-x11:x11-tools,resmgr,xkeyboard-config,xorg-x11-Mesa,libusb,freetype2,libjpeg,libpng
Ignore: apache2:logrotate
Ignore: arts:alsa,audiofile,resmgr,libogg,libvorbis
Ignore: kdelibs3:alsa,arts,pcre,OpenEXR,aspell,cups-libs,mDNSResponder,krb5,libjasper
Ignore: kdelibs3-devel:libvorbis-devel
Ignore: kdebase3:kdebase3-ksysguardd,OpenEXR,dbus-1,dbus-1-qt,hal,powersave,openslp,libusb
Ignore: kdebase3-SuSE:release-notes
Ignore: jack:alsa,libsndfile
Ignore: libxml2-devel:readline-devel
Ignore: gnome-vfs2:gnome-mime-data,desktop-file-utils,cdparanoia,dbus-1,dbus-1-glib,krb5,hal,libsmbclient,fam,file_alteration
Ignore: libgda:file_alteration
Ignore: gnutls:lzo,libopencdk
Ignore: gnutls-devel:lzo-devel,libopencdk-devel
Ignore: pango:cairo,glitz,libpixman,libpng
Ignore: pango-devel:cairo-devel
Ignore: cairo-devel:libpixman-devel
Ignore: libgnomeprint:libgnomecups
Ignore: libgnomeprintui:libgnomecups
Ignore: orbit2:libidl
Ignore: orbit2-devel:libidl,libidl-devel,indent
Ignore: qt3:libmng
Ignore: qt-sql:qt_database_plugin
Ignore: gtk2:libpng,libtiff
Ignore: libgnomecanvas-devel:glib-devel
Ignore: libgnomeui:gnome-icon-theme,shared-mime-info
Ignore: scrollkeeper:docbook_4,sgml-skel
Ignore: gnome-desktop:libgnomesu,startup-notification
Ignore: python-devel:python-tk
Ignore: gnome-pilot:gnome-panel
Ignore: gnome-panel:control-center2
Ignore: gnome-menus:kdebase3
Ignore: gnome-main-menu:rug
Ignore: libbonoboui:gnome-desktop
Ignore: postfix:pcre
Ignore: docbook_4:iso_ent,sgml-skel,xmlcharent
Ignore: control-center2:nautilus,evolution-data-server,gnome-menus,gstreamer-plugins,gstreamer,metacity,mozilla-nspr,mozilla,libxklavier,gnome-desktop,startup-notification
Ignore: docbook-xsl-stylesheets:xmlcharent
Ignore: liby2util-devel:libstdc++-devel,openssl-devel
Ignore: yast2:yast2-ncurses,yast2-theme-SuSELinux,perl-Config-Crontab,yast2-xml,SuSEfirewall2
Ignore: yast2-core:netcat,hwinfo,wireless-tools,sysfsutils
Ignore: yast2-core-devel:libxcrypt-devel,hwinfo-devel,blocxx-devel,sysfsutils,libstdc++-devel
Ignore: yast2-packagemanager-devel:rpm-devel,curl-devel,openssl-devel
Ignore: yast2-devtools:perl-XML-Writer,libxslt,pkgconfig
Ignore: yast2-installation:yast2-update,yast2-mouse,yast2-country,yast2-bootloader,yast2-packager,yast2-network,yast2-online-update,yast2-users,release-notes,autoyast2-installation
Ignore: yast2-bootloader:bootloader-theme
Ignore: yast2-packager:yast2-x11
Ignore: yast2-x11:sax2-libsax-perl
Ignore: openslp-devel:openssl-devel
Ignore: java-1_4_2-sun:xorg-x11-libs
Ignore: java-1_4_2-sun-devel:xorg-x11-libs
Ignore: kernel-um:xorg-x11-libs
Ignore: tetex:xorg-x11-libs,expat,fontconfig,freetype2,libjpeg,libpng,ghostscript-x11,xaw3d,gd,dialog,ed
Ignore: yast2-country:yast2-trans-stats
Ignore: susehelp:susehelp_lang,suse_help_viewer
Ignore: mailx:smtp_daemon
Ignore: cron:smtp_daemon
Ignore: hotplug:syslog
Ignore: pcmcia:syslog
Ignore: avalon-logkit:servlet
Ignore: jython:servlet
Ignore: ispell:ispell_dictionary,ispell_english_dictionary
Ignore: aspell:aspel_dictionary,aspell_dictionary
Ignore: smartlink-softmodem:kernel,kernel-nongpl
Ignore: OpenOffice_org-de:myspell-german-dictionary
Ignore: mediawiki:php-session,php-gettext,php-zlib,php-mysql,mod_php_any
Ignore: squirrelmail:mod_php_any,php-session,php-gettext,php-iconv,php-mbstring,php-openssl
Ignore: simias:mono(log4net)
Ignore: zmd:mono(log4net)
Ignore: horde:mod_php_any,php-gettext,php-mcrypt,php-imap,php-pear-log,php-pear,php-session,php
Ignore: xerces-j2:xml-commons-apis,xml-commons-resolver
Ignore: xdg-menu:desktop-data
Ignore: nessus-libraries:nessus-core
Ignore: evolution:yelp
Ignore: mono-tools:mono(gconf-sharp),mono(glade-sharp),mono(gnome-sharp),mono(gtkhtml-sharp),mono(atk-sharp),mono(gdk-sharp),mono(glib-sharp),mono(gtk-sharp),mono(pango-sharp)
Ignore: gecko-sharp2:mono(glib-sharp),mono(gtk-sharp)
Ignore: vcdimager:libcdio.so.6,libcdio.so.6(CDIO_6),libiso9660.so.4,libiso9660.so.4(ISO9660_4)
Ignore: libcdio:libcddb.so.2
Ignore: gnome-libs:libgnomeui
Ignore: nautilus:gnome-themes
Ignore: gnome-panel:gnome-themes
Ignore: gnome-panel:tomboy
Substitute: utempter
%ifnarch s390 s390x ppc ia64
Substitute: java2-devel-packages java-1_4_2-sun-devel
%else
 %ifnarch s390x
Substitute: java2-devel-packages java-1_4_2-ibm-devel
 %else
Substitute: java2-devel-packages java-1_4_2-ibm-devel xorg-x11-libs-32bit
 %endif
%endif
Substitute: yast2-devel-packages docbook-xsl-stylesheets doxygen libxslt perl-XML-Writer popt-devel sgml-skel update-desktop-files yast2 yast2-devtools yast2-packagemanager-devel yast2-perl-bindings yast2-testsuite
#
# SUSE compat mappings
#
Substitute: gcc-c++ gcc
Substitute: libsigc++2-devel libsigc++-2.0-dev
Substitute: glibc-devel-32bit
Substitute: pkgconfig pkg-config
%ifarch %ix86
Substitute: kernel-binary-packages kernel-default kernel-smp kernel-bigsmp kernel-debug kernel-um kernel-xen kernel-kdump
%endif
%ifarch ia64
Substitute: kernel-binary-packages kernel-default kernel-debug
%endif
%ifarch x86_64
Substitute: kernel-binary-packages kernel-default kernel-smp kernel-xen kernel-kdump
%endif
%ifarch ppc
Substitute: kernel-binary-packages kernel-default kernel-kdump kernel-ppc64 kernel-iseries64
%endif
%ifarch ppc64
Substitute: kernel-binary-packages kernel-ppc64 kernel-iseries64
%endif
%ifarch s390
Substitute: kernel-binary-packages kernel-s390
%endif
%ifarch s390x
Substitute: kernel-binary-packages kernel-default
%endif
%define debian_version 800
Macros:
%debian_version 800
Visit webUI to check project configuration Create an OBS project linked to DoD
$ osc -A https://stretch:443 meta prj test -e
<project name= test >
<title>test</title>
<description>test</description>
<person userid= Admin role= maintainer />
<repository name= Debian_8.0 >
<path project= Debian:8 repository= main />
<arch>x86_64</arch>
</repository>
</project>
Visit webUI to check project configuration Adding a package to the project
$ osc -A https://stretch:443 co test ; cd test
$ mkdir hello ; cd hello ; apt-get source -d hello ; cd - ; 
$ osc add hello 
$ osc ci -m "New import" hello
The package should go to dispatched state then get in blocked state while it downloads build dependencies from DoD link, eventually it should start building. Please check the journal logs to check if something went wrong or gets stuck. Visit webUI to check hello package build state OBS logging to the journal Check in the journal logs everything went fine:
$ sudo journalctl -u obsdispatcher.service -u obsdodup.service -u obsscheduler@x86_64.service -u obsworker.service -u obspublisher.service
Troubleshooting Currently we are facing few issues with web UI: And there are more issues that have not been reported, please do reportbug obs-api .

18 October 2016

Dirk Eddelbuettel: gettz 0.0.2

Release 0.0.2 of gettz is now on CRAN. gettz provides a possible fallback in situations where Sys.timezone() fails to determine the system timezone. That can happen when e.g. the file /etc/localtime somehow is not a link into the corresponding file with zoneinfo data in, say, /usr/share/zoneinfo. Windows is now no longer excluded, though it doesn't do anything useful yet. The main use of the package is still for Linux. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

30 September 2016

Chris Lamb: Free software activities in September 2016

Here is my monthly update covering what I have been doing in the free software world (previous month):
Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws, most Linux distributions provide binary (or "compiled") packages to end users. The motivation behind the Reproducible Builds effort is to allow verification that no flaws have been introduced either maliciously and accidentally during this compilation process by promising identical binary packages are always generated from a given source. My work in the Reproducible Builds project was also covered in our weekly reports #71, #72, #71 & #74. I made the following improvements to our tools:

diffoscope

diffoscope is our "diff on steroids" that will not only recursively unpack archives but will transform binary formats into human-readable forms in order to compare them.

  • Added a global Progress object to track the status of the comparison process allowing for graphical and machine-readable status indicators. I also blogged about this feature in more detail.
  • Moved the global Config object to a more Pythonic "singleton" pattern and ensured that constraints are checked on every change.

disorderfs

disorderfs is our FUSE filesystem that deliberately introduces nondeterminism into the results of system calls such as readdir(3).

  • Display the "disordered" behaviour we intend to show on startup. (#837689)
  • Support relative paths in command-line parameters (previously only absolute paths were permitted).

strip-nondeterminism

strip-nondeterminism is our tool to remove specific information from a completed build.

  • Fix an issue where temporary files were being left on the filesystem and add a test to avoid similar issues in future. (#836670)
  • Print an error if the file to normalise does not exist. (#800159)
  • Testsuite improvements:
    • Set the timezone in tests to avoid a FTBFS and add a File::StripNondeterminism::init method to the API to to set tzset everywhere. (#837382)
    • "Smoke test" the strip-nondeterminism(1) and dh_strip_nondeterminism(1) scripts to prevent syntax regressions.
    • Add a testcase for .jar file ordering and normalisation.
    • Check the stripping process before comparing file attributes to make it less confusing on failure.
    • Move to a lookup table for descriptions of stat(1) indices and use that for nicer failure messages.
    • Don't uselessly test whether the inode number has changed.
  • Run perlcritic across the codebase and adopt some of its prescriptions including explicitly using oct(..) for integers with leading zeroes, avoiding mixing high and low-precedence booleans, ensuring subroutines end with a return statement, etc.

I also submitted 4 patches to fix specific reproducibility issues in golang-google-grpc, nostalgy, python-xlib & torque.


Debian https://lamby-www.s3.amazonaws.com/yadt/blog.Image/image/original/28.jpeg

Patches contributed

Debian LTS

This month I have been paid to work 12.75 hours on Debian Long Term Support (LTS). In that time I did the following:
  • "Frontdesk" duties, triaging CVEs, etc.
  • Issued DLA 608-1 for mailman fixing a CSRF vulnerability.
  • Issued DLA 611-1 for jsch correcting a path traversal vulnerability.
  • Issued DLA 620-1 for libphp-adodb patching a SQL injection vulnerability.
  • Issued DLA 631-1 for unadf correcting a buffer underflow issue.
  • Issued DLA 634-1 for dropbear fixing a buffer overflow when parsing ASN.1 keys.
  • Issued DLA 635-1 for dwarfutils working around an out-of-bounds read issue.
  • Issued DLA 638-1 for the SELinux policycoreutils, patching a sandbox escape issue.
  • Enhanced Brian May's find-work --unassigned switch to take an optional "except this user" argument.
  • Marked matrixssl and inspircd as being unsupported in the current LTS version.

Uploads
  • python-django 1:1.10.1-1 New upstream release and ensure that django-admin startproject foo creates files with the correct shebang under Python 3.
  • gunicorn:
    • 19.6.0-5 Don't call chown(2) if it would be a no-op to avoid failure under snap.
    • 19.6.0-6 Remove now-obsolete conffiles and logrotate scripts; they should have been removed in 19.6.0-3.
  • redis:
    • 3.2.3-2 Call ulimit -n 65536 by default from SysVinit scripts to normalise the behaviour with systemd. I also bumped the Debian package epoch as the "2:" prefix made it look like we are shipping version 2.x. I additionaly backported this upload to Debian Jessie.
    • 3.2.4-1 New upstream release, add missing -ldl for dladdr(3) & add missing dependency on lsb-base.
  • python-redis (2.10.5-2) Bump python-hiredis to Suggests to sync with Ubuntu and move to a machine-readable debian/copyright. I also backported this upload to Debian Jessie.
  • adminer (4.2.5-3) Move mysql-server dependencies to default-mysql-server. I also backported this upload to Debian Jessie.
  • gpsmanshp (1.2.3-5) on behalf of the QA team:
    • Move to "minimal" debhelper style, making the build reproducible. (#777446 & #792991)
    • Reorder linker command options to build with --as-needed (#729726) and add hardening flags.
    • Move to machine-readable copyright file, add missing #DEBHELPER# tokens to postinst and prerm scripts, tidy descriptions & other debian/control fields and other smaller changes.

I sponsored the upload of 5 packages from other developers:

I also NMU'd:



FTP Team

As a Debian FTP assistant I ACCEPTed 147 packages: alljoyn-services-1604, android-platform-external-doclava, android-platform-system-tools-aidl, aufs, bcolz, binwalk, bmusb, bruteforce-salted-openssl, cappuccino, captagent, chrome-gnome-shell, ciphersaber, cmark, colorfultabs, cppformat, dnsrecon, dogtag-pki, dxtool, e2guardian, flask-compress, fonts-mononoki, fwknop-gui, gajim-httpupload, glbinding, glewmx, gnome-2048, golang-github-googleapis-proto-client-go, google-android-installers, gsl, haskell-hmatrix-gsl, haskell-relational-query, haskell-relational-schemas, haskell-secret-sharing, hindsight, i8c, ip4r, java-string-similarity, khal, khronos-opencl-headers, liblivemedia, libshell-config-generate-perl, libshell-guess-perl, libstaroffice, libxml2, libzonemaster-perl, linux, linux-grsec-base, linux-signed, lua-sandbox, lua-torch-trepl, mbrola-br2, mbrola-br4, mbrola-de1, mbrola-de2, mbrola-de3, mbrola-ir1, mbrola-lt1, mbrola-lt2, mbrola-mx1, mimeo, mimerender, mongo-tools, mozilla-gnome-keyring, munin, node-grunt-cli, node-js-yaml, nova, open-build-service, openzwave, orafce, osmalchemy, pgespresso, pgextwlist, pgfincore, pgmemcache, pgpool2, pgsql-asn1oid, postbooks-schema, postgis, postgresql-debversion, postgresql-multicorn, postgresql-mysql-fdw, postgresql-unit, powerline-taskwarrior, prefix, pycares, pydl, pynliner, pytango, pytest-cookies, python-adal, python-applicationinsights, python-async-timeout, python-azure, python-azure-storage, python-blosc, python-can, python-canmatrix, python-chartkick, python-confluent-kafka, python-jellyfish, python-k8sclient, python-msrestazure, python-nss, python-pytest-benchmark, python-tenacity, python-tmdbsimple, python-typing, python-unidiff, python-xstatic-angular-schema-form, python-xstatic-tv4, quilt, r-bioc-phyloseq, r-cran-filehash, r-cran-png, r-cran-testit, r-cran-tikzdevice, rainbow-mode, repmgr, restart-emacs, restbed, ruby-azure-sdk, ruby-babel-source, ruby-babel-transpiler, ruby-diaspora-prosody-config, ruby-haikunator, ruby-license-finder, ruby-ms-rest, ruby-ms-rest-azure, ruby-rails-assets-autosize, ruby-rails-assets-blueimp-gallery, ruby-rails-assets-bootstrap, ruby-rails-assets-bootstrap-markdown, ruby-rails-assets-emojione, ruby-sprockets-es6, ruby-timeliness, rustc, skytools3, slony1-2, snmp-mibs-downloader, syslog-ng, test-kitchen, uctodata, usbguard, vagrant-azure, vagrant-mutate & vim.

20 September 2016

Reproducible builds folks: Reproducible Builds: week 73 in Stretch cycle

What happened in the Reproducible Builds effort between Sunday September 11 and Saturday September 17 2016: Toolchain developments Ximin Luo started a new series of tools called (for now) debrepatch, to make it easier to automate checks that our old patches to Debian packages still apply to newer versions of those packages, and still make these reproducible. Ximin Luo updated one of our few remaining patches for dpkg in #787980 to make it cleaner and more minimal. The following tools were fixed to produce reproducible output: Packages reviewed and fixed, and bugs filed The following updated packages have become reproducible - in our current test setup - after being fixed: The following updated packages appear to be reproducible now, for reasons we were not able to figure out. (Relevant changelogs did not mention reproducible builds.) The following 3 packages were not changed, but have become reproducible due to changes in their build-dependencies: jaxrs-api python-lua zope-mysqlda. Some uploads have addressed some reproducibility issues, but not all of them: Patches submitted that have not made their way to the archive yet: Reviews of unreproducible packages 462 package reviews have been added, 524 have been updated and 166 have been removed in this week, adding to our knowledge about identified issues. 25 issue types have been updated: Weekly QA work FTBFS bugs have been reported by: diffoscope development A new version of diffoscope 60 was uploaded to unstable by Mattia Rizzolo. It included contributions from: It also included from changes previous weeks; see either the changes or commits linked above, or previous blog posts 72 71 70. strip-nondeterminism development New versions of strip-nondeterminism 0.027-1 and 0.028-1 were uploaded to unstable by Chris Lamb. It included contributions from: disorderfs development A new version of disorderfs 0.5.1 was uploaded to unstable by Chris Lamb. It included contributions from: It also included from changes previous weeks; see either the changes or commits linked above, or previous blog posts 70. Misc. This week's edition was written by Ximin Luo and reviewed by a bunch of Reproducible Builds folks on IRC.

10 September 2016

Dirk Eddelbuettel: New package gettz on CRAN

gettz is now on CRAN in its initial release 0.0.1. It provides a possible fallback in situations where Sys.timezone() fails to determine the system timezone. That can happen when e.g. the file /etc/localtime somehow is not a link into the corresponding file with zoneinfo data in, say, /usr/share/zoneinfo. Duane McCully provided a nice StackOverflow answer with code that offers fallbacks via /etc/timezone (on Debian/Ubuntu) or /etc/sysconfig/clock (on RedHat/CentOS/Fedora, and rumour has it, BSD* systems) or /etc/TIMEZONE (on Solaris). The gettz micro-package essentially encodes that approach so that we have an optional fallback when Sys.timezone() comes up empty. In the previous paragraph, note the stark absense of OS X where there seems nothing to query, and of course Windows. Contributions for either would be welcome. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

3 September 2016

Antonio Terceiro: testing build reprodubility with debrepro

Earlier today I was handling a reproducibility bug and decided I had to try a reproducibility test by myself. I tried reprotest, but I was being hit by a disorderfs issue and I was not sure whether the problem was with reprotest or not (at this point I cannot reproduce that anymore). So I decided to hack a simple script to that, and it works. I even included it in devscripts after writing a manpage. Of course reprotest is more complete, extensible, and supports arbitrary virtualization backends for doing the more dangerous/destructive variations (such as changing the hostname and other things that require root) but for quick tests debrepro does the job. Usage examples:

$ debrepro                                 # builds current directory
$ debrepro /path/to/sourcepackage          # builds package there
$ gbp-buildpackage --git-builder=debrepro  # can be used with vcs wrappers as well

debrepro will do two builds with a few variations between them, including $USER, $PATH, timezone, locale, umask, current time, and will even build under disorderfs if available. Build path variation is also performed because by definition the builds are done in different directories. If diffoscope is installed, it will be used for deep comparison of non-matching binaries. If you are interested and don t want to build devscripts from source or wait for the next release, you can just grab the script, save it as debrepro somewhere on your $PATH and make it executable.

Next.

Previous.