Search Results: "Jonathan Yu"

5 May 2013

Vasudev Kamath: CDBS Packaging: package relationship management

(It took me a while to come up with new CDBS packaging series post not because I stopped using CDBS just because I was procrastinating myself as busy) This is the second post in the CDBS packaging series. In this series I'm going to talk about package relationship management. The better example where this feature is useful is packages where build-depends and run-time dependencies overlap. Most of the Perl modules which have test suites have build-depend and run-time dependency intersection. So let me take example of a Perl module. First lets see control file of a Perl package which is not using CDBS and then let me explain how CDBS can help you improve the situation. I choose libxml-libxml-perl lets see part of control file which includes Build-Depends Depends Suggests Recommends.
Source: libxml-libxml-perl
Maintainer: Debian Perl Group <pkg-perl-maintainers@lists.alioth.debian.org>
Uploaders: Jonathan Yu <jawnsy@cpan.org>,
 gregor herrmann <gregoa@debian.org>,
 Chris Butler <chrisb@debian.org>
Section: perl
Priority: optional
Build-Depends: perl (>= 5.12),
 debhelper (>= 9.20120312),
 libtest-pod-perl,
 libxml2-dev,
 libxml-namespacesupport-perl,
 libxml-sax-perl,
 zlib1g-dev
Standards-Version: 3.9.4
Vcs-Browser: http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libxml-libxml-perl.git
Vcs-Git: git://anonscm.debian.org/pkg-perl/packages/libxml-libxml-perl.git
Homepage: https://metacpan.org/release/XML-LibXML/
Package: libxml-libxml-perl
Architecture: any
Depends: $ shlibs:Depends , $ misc:Depends , $ perl:Depends ,
 libxml-namespacesupport-perl,
 libxml-sax-perl
Breaks: libxml-libxml-common-perl
Replaces: libxml-libxml-common-perl
Description: Perl interface to the libxml2 library
 XML::LibXML is a Perl interface to the GNOME libxml2 library, which provides
 interfaces for parsing and manipulating XML files. This module allows Perl
 programmers to make use of the highly capable validating XML parser and the
 high performance Document Object Model (DOM) implementation. Additionally, it
 supports using the XML Path Language (XPath) to find and extract information.
So 2 packages are both in Build-Depends and Depends field
  1. libsax-xml-perl
  2. libxml-namespacesupport-perl
So in this situation there is a possibility that we miss to add one or both of these packages into Depends field, I'm not saying we will surely miss but we might after all we are all human beings. So how can we improve the situation using CDBS? Let me go through step by step on what we need to do.
  1. Create a file called control.in with same above contents but slight modification in Build-Depends and Depends section. I will display the diff below to avoid re-pasting entire file again and again.
--- debian/control      2013-04-28 23:08:11.930082600 +0530
+++ debian/control.in   2013-05-04 20:51:18.849680419 +0530
@@ -5,13 +5,7 @@
  Chris Butler <chrisb@debian.org>
 Section: perl
 Priority: optional
-Build-Depends: perl (>= 5.12),
- debhelper (>= 9.20120312),
- libtest-pod-perl,
- libxml2-dev,
- libxml-namespacesupport-perl,
- libxml-sax-perl,
- zlib1g-dev
+Build-Depends: @cdbs@
 Standards-Version: 3.9.4
 Vcs-Browser: http://anonscm.debian.org/gitweb/?p=pkg-perl/packages/libxml-libxml-perl.git
 Vcs-Git: git://anonscm.debian.org/pkg-perl/packages/libxml-libxml-perl.git
@@ -20,8 +14,7 @@
 Package: libxml-libxml-perl
 Architecture: any
 Depends: $ shlibs:Depends , $ misc:Depends , $ perl:Depends ,
- libxml-namespacesupport-perl,
- libxml-sax-perl
+ $ cdbs:Depends 
 Breaks: libxml-libxml-common-perl
 Replaces: libxml-libxml-common-perl
 Description: Perl interface to the libxml2 library
@@ -30,4 +23,3 @@
  programmers to make use of the highly capable validating XML parser and the
  high performance Document Object Model (DOM) implementation. Additionally, it
  supports using the XML Path Language (XPath) to find and extract information.
-
  1. Next we need to have something like below in the rule files
#!/usr/bin/make -f
include /usr/share/cdbs/1/rules/debhelper.mk
include /usr/share/cdbs/1/rules/utils.mk
include /usr/share/cdbs/1/rules/upstream-tarball.mk
include /usr/share/cdbs/1/class/perl-makemaker.mk
pkg = $(DEB_SOURCE_PACKAGE)
deps = libxml-libxml-perl libxml-sax-perl
deps-test = libtest-pod-perl
CDBS_BUILD_DEPENDS +=, $(deps), $(deps-test)
CDBS_BUILD_DEPENDS +=, zlib1g-dev, libxml2-dev, perl (>= 5.12)
CDBS_DEPENDS_$(pkg) = , $(deps)
So basically we moved all Build-Depends and Depends to rules file. The common ones are placed in deps variable and assigned to both Build-Depends and Depends. CDBS uses following variables for package relationship management.
  1. CDBS_BUILD_DEPENDS: This variable helps you manage Build-Depends field and all you need to do is in your control.in files Build-Depends field put place holder @cdbs@
  2. CDBS_DEPENDS: This field can be used to manage Depends field of binary package for each binary package you need to have one CDBS_DEPENDS_pkgname variable with depends assigned to it. And in your control.in append $ cdbs:Depends to Depends field.
  3. CDBS_PROVIDES, CDBS_BREAKS, CDBS_RECOMMENDS, CDBS_PREDEPENDS, CDBS_SUGGESTS, CDBS_REPLACES: all these does the job what you think it does :-).
Other than CDBS_BUILD_DEPENDS all other variables work using substvars i.e. CDBS will put the respective substitutions in pkgname.substvars file which will be used during deb creation to replace things in control file. So to make CDBS generate new control file run the below command
DEB_MAINTAINER_MODE=1 fakeroot debian/rules debian/control
Basically this command needs to be executed before starting build process if you miss your changes will not be reflected into debian/control. Additionally this feature is Maintainer Mode helper tool because Debian policy prohibits change of debian/control during normal package build. So what is the benefit of using this feature of CDBS? I've listed some of them which I felt are obvious.
  1. When there is intersection in Build-Depends and Depends this feature from CDBS is helpful. As I shown above put all intersecting dependencies in common variable and appropriately assign them wherever it is required. Thus we can avoid possible missing of some of run-time dependencies due to human error.
  2. It is also possible that newer version of package requires specific version of some package (mostly libraries) and we updated build dependencies but we might forget to do the same in Depends, by using this feature we can make sure we will not miss such minute details.
One last thing I want to point out is if you are NMUing a CDBS package
NMUs need not (but are encouraged to) make special use of these tools. In particular, the debian/control.in file can be completely ignored.
Before closing down the post, If you find some mistake in the post please let me know either through comments or through the email. Soon I will be back with new CDBS recipes till then cya.

12 February 2012

Gregor Herrmann: RC bugs 2012/05-06

I was at FOSDEM over the last weekend, so here's my report for two weeks now:

6 April 2010

Jonathan Yu: Google Summer of Code 2010

Last year, I had a great time participating in the Google Summer of Code with the Debian project. I had a neat project with some rather interesting implications for helping developers to package and maintain their work. It s still a work-in-progress, of course, as many projects in open source are, but I was able to accomplish quite a bit and am proud of my work. I learned quite a bit about coding in C, working with Debian and met some very intelligent people. My student peers were also very intelligent and great to learn from. I enjoyed meeting them virtually and discussing our various projects on the IRC channel as the summer progressed and the Summer of Code kicked into full swing. The Debian project in particular also helps arrange travel grants for students to attend the Debian Conference (this year, DebConf10 is being held in New York City!). DebConf provides a great venue to learn from other developers (both in official talks but also unofficial hacking sessions). As the social aspect is particularly important to Debian, DebConf helps people meet those with whom they are working with the most, thereby creating lifelong friendships and making open source fun. I have had several interviews for internships, and the bit of my work experience most asked about is my time doing the Google Summer of Code. I really enjoyed seeing a project go from the proposal stage, setting a reasonable timeline with my mentor, exploring the state of the art, and most importantly, developing the software. I think this is the sort of indispensible industry-type experience we often lack in our undergrad education. We might have an honours thesis or presentation, but much of the work in the Google Summer of Code actually gets used in the field. Developing software for people rather than for marks is significant in a number of ways, but most importantly it means there are real stakeholders that must be considered at all stages. Proposing brilliant new ideas is important, however, without highlighting the benefits they can have for various users, the reality is that it simply will not gain traction. Learning how to write proposals effectively is an important skill and working with my prospective mentor (at the time he later mentored my project once it was accepted) to develop mine was tremendously useful for my future endeavours. The way I see the Google Summer of Code is, in many ways, similar to an academic grant (and the stipend is about the same as well). It provides a modest salary (this year it s US$5000) but more importantly, personal contact with a mentor. Mentors are typically veterans in software development or the Debian project and act in the same role as supervisors for post-graduate work: they help monitor your progress and propose new ideas to keep you on track. The Debian Project is looking for more students and proposals. We have a list of ideas as well as application instructions available on our Wiki. As I will be going on internship starting May, I have offered to be a mentor this year. I look forward to seeing your submissions (some really interesting ones have already begun to filter in as the deadline approaches).
Filed under: Computer Science, Peer Relationships, Software Engineering Tagged: Computer Science, Debian, Google Summer of Code, Support

27 May 2009

Jonathan Yu: I Love Open Source


While working on my Google Summer of Code project today, I came across a bug that pretty much halted my productivity for the day. Early on in my project, I decided that working with Unicode is hard, among other things. Since I was restricted to using C, I had to find a way to easily manipulate Unicode stuff, and I came across GLib (I m not even entirely sure how, I think I just remember other projects using it, and decided to look it up.) Not only did it have Unicode handling stuff, it also provides a bunch of convenient things like a linked list implementation, memory allocation, etc. All in a way intended to be cross-platform compatible, since this is the thing that s used to power Gtk. I m not entirely sure how it differs from Apache s Portable Runtime (APR); maybe it s even a Not Invented Here syndrome. In any case, I, not suffering from that particular affliction, decided to be lazy and re-use existing code. For some reason, GLib s g_slice_alloc() call was failing. For those of you that don t know what this does, it is similar to malloc() in standard C it allocates a chunk of memory and returns it to you, so that you can make use of dynamic memory allocation, rather than everything just being auto variables. In particular, it means you can be flexible and allocate as much or as little memory as you need. So I spent the entire day trying to figure out why my program was segfaulting. Looking at the output of gdb (the GNU Debugger), the backtrace showed that it was crashing at the allocation statement. No way, I thought, that test is so well-tested, it must be a problem with the way I m using it. I changed the code to use malloc() instead of g_slice_alloc(), and the program started crashing right away, rather than after four or five executions with g_slice_alloc(). Not exactly useful for debugging. I mentioned my frustration with C on the Debian Perl Packager Group channel, and a friend from the group, Ryan Niebur took a look at the code (accessible via a public repository). After a bit of tinkering, he determined that the problem was that I was using g_slice_alloc instead of g_slice_alloc0, which automatically zeroes memory before returning it. It stopped the crashing and my program works as intended. I m still left totally puzzled as to why this bug was happening, and I m not sure if malloc isn t supposed to be used with structs, or some other limitation like that. But thanks the magic of open source and social coding/debugging, the expertise of many contribute to the success of a single project. It s such a beautiful thing. Posted in Computer Science, Software Engineering Tagged: Algorithms, Best Practices, Code Reuse, Computer Science, Google Summer of Code

10 April 2009

Obey Arthur Liu: Google Summer of Code 2009: Debian s Shortlist

Copy of http://lists.debian.org/debian-devel/2009/04/msg00421.html. Hi folks, We have been pretty busy these past few weeks with the whole Google Summer of Code 2009 student application process.
I can say that we have this year a very good set of proposals and I d like to thank all the students and mentors for this. I am going to present to you our shortlist of projects that we would like to be funded and believe we can reasonably manage to get funded. As always, remember that the number of slots is not final yet at this point so we can t promise anything. The first preliminary slot count given today was *10* (same as last year) and we hope to get *2* more (as we did last year). This shortlist is alphabetically ordered because we don t want to reveal the current internal rankings. I am inviting you to debate what you think is cool, what is useful, what is important to Debian, maybe give us pointers to resources or people that could be helpful for the projects. We will try to alter our current rankings to reflect the zeitgeist in Debian, while taking into account the personal information that we have about each student involved. The deadline for any modification is on the 15th, so get everything in by the 14th. The final selected projects will be announced by Google April 20th, ~12 noon PDT / 19:00 UTC. We ll have another announcement then. Three proposals need or may need a mentor, I indicated it. For more information about the projects or mentoring and how to talk to us directly, scroll down past the list. Debian s Shortlist : - Aptitude Package Management History Tracking
- Automatic Debug Packages Creation and Handling
- Debbugs Web UI: Amancay Strikes Back
- Control Files Parsing/Editing Library/Qt4-Debconf Qt4-Perl bindings
- Debian-Installer Support for GNU/kFreeBSD
- KDE/Qt4 Adept 3.0 Package Manager
- Large Scientific Dataset Package Management
- MIPS N32 ABI Port
- MTD Embedded Onboard flash Partitioning and Installation
- On-demand Cloud Computing with Amazon EC2 and Eucalyptus Integration
- Port back update-manager to Debian and all Derivatives
- Debian Autobuilding Infrastructure Rewrite And the details: Aptitude Package Management History Tracking Student: Cristian Mauricio Porras Duarte, Mentor: Daniel Burrows Aptitude currently does not track actions that the user has performed beyond a single session of the program. One of the most frequent requests from users is to find out when they made a change to a package, or why a package was changed; we want to store this information and expose it in the UI in convenient locations. As a side effect, this might also provide some ability to revert past changes. Automatic Debug Packages Creation and Handling Student: Emilio Pozuelo Monfort, Mentor: Marc Brockschmidt This proposal aims at providing debug binary packages for the packages in the Debian archive in an automatic manner, moving them away from the official Debian archive to an special one. This has the benefits of providing thousands of debug packages without any work needed from the developers, for all the architectures, without bloating
the archive. Debbugs Web UI: Amancay Strikes Back Student: Diego Escalante Urrelo, Mentor: Margarita Manterola The Amancay project aims to be a new read/write web frontend to Debian s BTS; allowing DDs and contributors to easily interact with bugs via an intuitive yet powerful interface, enabling new workflows and creating new contribution opportunities like triaging while upholding reporting quality. Control Files Parsing/Editing Library/Qt4-Debconf Qt4-Perl bindings Student: Jonathan Yu, Mentor: (probably) Dominique Dumont see below This project proposes a common library for parsing and manipulating Debian Control files, including control, copyright and changelog. Main ideas include validating and parsing of these files, with both Strict and Quirks modes for the parser. The second idea is a new frontend for Debconf using Qt4 (for which Perl bindings will be written). Debian-Installer Support for GNU/kFreeBSD Student: Luca Favatella, Mentor: Aurelien Jarno GNU/kFreeBSD is currently using a hacked version of the FreeBSD installer combined with crosshurd as its own installer. While this works more or less correctly for standard installations (read: the exact same installation as in the documentation), it does not allow any changes in the installation process except the hard disk partitioning. This project is about porting debian-installer on GNU/kFreeBSD, and to a bigger extent, make debian-installer less Linux dependant. KDE/Qt4 Adept 3.0 Package Manager Student: Mateusz Marek, Mentor: NEEDS MENTOR, see below. Finish Adept 3.0, a fully integrated package manager for Qt4/KDE4. Adept is currently the only viable path to a Debian native package manager on KDE that would support modern features such as tags, indexed search or good conflict resolving. With Aptitude-gtk still in development and only available for GTK+ and (K)PackageKit having fundamental problems, Debian needs this project to stay in control of its package management on KDE after much neglect in the recent years. Large Scientific Dataset Package Management Student: Roy Flemming Hvaara, Mentor: Charles Plessy Large public datasets, like databases for bioinformatics are typically too big and too volatile to fit the traditional source/binary packaging scheme of Debian. There are some programs that are distributed in Debian, like blast and emboss, that can index specialised databases, but Debian lacks a tool to install or update the datasets they need and keep their indexing in sync. MIPS N32 ABI Port Student: Sha Liu, Mentor: Anthony Fok This project first focuses on creating a new MIPS N32 ABI port for Debian. Different from O32 and N64, N32 is an address model which has most 64-bit capabilities but using 32-bit data structures to save space and process time. A second focus will be given on making such a mipsn32el arch fully optimized for the Loongson 2F CPU which gains more and more popularity in subnotebooks/netbooks in many countries. MTD Embedded Onboard flash Partitioning and Installation Student: Per Andersson, Mentor: Wookey Many embedded devices have MTD onboard flash as persistent storage like the Kurobox Pro NAS, the Neo Freerunner, the Sheeva Plug or the OLPC. With MTD flash being so popular and with increases in capacity, support for MTD partition/installation would make Debian even more interesting to a wide range of of devices, making it one step closer to being universal. On-demand Cloud Computing with Amazon EC2 and Eucalyptus Integration Student: David Wendt Jr, Mentor: (probably) Steffen Moeller see below In many academic fields, as well as commercial industries, people use clusters to distribute tasks among multiple machines. Many times this is done by packaging a whole operating system disk image, uploading it onto the cluster, and having the cluster run it in a VM. This project intends to make it easier for Debian to distribute prepared disk images templates like they distribute CD images now, for the users to recreate or customise these templates with Debian packages and for administrators to host such clusters with Debian. Port back update-manager to Debian and all Derivatives Student: Stephan Peijnik, Mentor: Michael Vogt The project would involve taking the distribution-(Ubuntu-)specific update-manager code, analyzing it, and creating a package with just its core functionality, decoupling the distribution-specific parts and thus making the core code extensible by distribution-specific add-ons. This in turn would remove the need of porting update-manager to Debian with every upstream release. An additional optional goal would be replacing the synaptics-backend with a python-apt based one. Debian Autobuilding Infrastructure Rewrite Student: Philipp Kern, Mentor: Luk Claes Rewrite the software that currently runs the Debian autobuilding infrastructure in a way that makes it more maintainable and robust. It will use Python as its programming language and PostgreSQL for the database backend. By harmonizing buildds, many build failures can be prevented and wasteful workload on buildd volunteers can be reduced. On mentoring: Petr Rockai, the original developer of Adept has offered help to anyone willing to adopt Adept. Sune Vuorela has offered help for any Qt4 and KDE related issues. *We really need a mentor here*. The student is quite competent but Google dictates that we provide a mentor to handle student management. Dominique Dumont, although not DD, has signaled interest in mentoring this, although it hasn t been confirmed yet. Sune Vuorela has offered to help co-mentor for the Qt4-Debconf and Qt4-Perl bindings part. Steffen Moeller has signaled interest in mentoring this, although it hasn t been formally confirmed yet. Charles Plessy of the Debian Med team will provide help for use cases related issues. Eric Hammond, developer of the original vmbuilder image creation tool and maintainer of a set of Debian and Ubuntu images will provide help for Amazon EC2 and image creation issues. Chris Grzegorczyk from the Eucalyptus team will provide help for Eucalyptus and Eucalyptus/Debian integration issues. Contacting us: Considering the tight schedule, most stuff happens live on IRC: #debian-soc on irc.debian.org You can also consult our wiki page for some additional information:
<http://wiki.debian.org/SummerOfCode2009> We have a mailing-list at:
<http://lists.alioth.debian.org/mailman/listinfo/soc-coordination> Keep this discussion on debian-devel@lists.debian.org while cc-ing soc-coordination@lists.alioth.debian.org. This thread is for debian-devel primarily.