Clint Adams: Forward the foundation
I don't know how I feel about this.
I don't know how I feel about this.
I don't know how I feel about this.
I don't know how I feel about this.
Maximilian, along with the other members of the Debian kernel team, has the overwhelming job of maintaining the Linux kernel in Debian. It s one of the largest package and certainly one where dealing with bug reports is really difficult as most of them are hardware-specific, and thus difficult to reproduce.
He s very enthusiastic and energetic, and does not fear criticizing when something doesn t please him. You ll see.
My questions are in bold, the rest is by Maximilian.
Who are you?
My name is Maximilian
Attems. I am a theoretical physicist in my last year of PhD at the
Technical University of Vienna. My main research area is the early phase of a
Quark-Gluon Plasma as produced in heavy ion collisions at the LHC at CERN. I am
developing simulations that take weeks on the
Vienna
Scientific Cluster (in the TOP 500 list). The rest of the lab is much less fancy and boils down
to straight intel boxes without any binary blobs or external drivers (although
lately we add radeon graphics for decent free 3D). Mathematica and Maple are
the rare exceptions to the many dev tools of Debian (LaTeX, editors, git,
IDE s, Open MPI, ..) found at the institute, as those are unfortunately yet
unmatched in Free Software for symbolic computations. The lab mostly runs a
combination of Debian stable (testing starting from freeze) for desktops and
oldstable/stable for servers. Debian is in use for more than 10 years. So
people in the institute know some ups and downs of the project. Newcomers like
my room neighbors are always surprised how functional a free Debian Desktop
is. 
What s your biggest achievement within Debian?
Building lots and lots of kernels together with an growing uptake of the officially released linux images.
I joined the Debian kernel team shortly after Herbert Xu departed. I had been
upstream Maintainer of the linux-2.6 janitor project for almost a year brewing
hundreds of small cleanups with quilt in a tree named
kjt for
early
linux-2.6. In Debian we had lots
of fun in sorting out the troubles that the long 2.5 freeze had imposed:
Meaning we were sitting on a huge diverging monolithic semi-good patchset.
It was great fun to prepare 2.6.8 for Sarge with a huge team enthusiastic in
shipping something real close to mainline (You have to imagine that back then
you had no stable or longterm release nor any useful free tools like git. This
involved passing patches around, hand editing them and seeing what the result
does.)
From the Sarge install reports a common pattern emerged that the current Debian
early userspace was causing lots of boot failures. This motivated
me to develop an alternative using the new upstream initramfs features. So I got
involved in early userspace. Thanks to large and active development team
initramfs-tools got a nice ecosystem. It still tries to be as generic and
flexible as possible and thus gains many nice features. Also H. Peter Anvin
(hpa) gave me the official
co-maintenance of klibc. klibc saw uptake and good patches from Google
in the last 2 years. I am proud that the early userspace is working out
fairly well these days, meaning you can shuffle discs around and see your box
boot.
Later on we focused on 2.6.18 for Etch, which turned out to a be good release
and picked up by several other distributions. Only very much later
we would see such a sync again. With 2.6.26 for Lenny we got somehow
unlucky as we just missed the new longterm release by one release.
We also pushed for another update very late (during freeze) in the release
cycle, which turned out to semi-work as too much things depend on linux-2.6.
For Squeeze 2.6.32 got picked thanks to discussions at Portland Linux Plumbers
and it turned out to be a good release picked up by many distributions and
external patchsets. The long-term support is going very well. Greg KH is doing
a great job in collecting various needed fixes for it. Somehow we had hoped
that the Squeeze freeze would start sooner and that the freeze duration would be
shorter, since we were ready for a release starting from the actual freeze on.
The only real big bastard on the cool 2.6.32 sync is Red Hat. Red Hat
Enterprise 6.0 is shipping the linux-2.6 2.6.32 in obfuscated form. They
released their linux-2.6 as one big tarball clashing with the spirit of the
GPL. One can only mildly guess from the changelog which patches get applied.
This is in sharp contrast to any previous Red Hat release and has not yet
generated the sharp and snide comments in press it deserves. Red Hat should
really step back and not make such stupid management moves. Next to them
even the semi-maintained Oracle Unbreakable 2.6.32 branch looks better:
It is git fetchable.
What are your plans and those of the kernel team for Debian Wheezy?
Since 2.6.32 many of the used patches landed upstream or are on the way
(speakup, Kbuild Debian specific targets, ..). The proper vfs based unionfs is
something we d be looking forward. We haven t yet picked the next upstream
release we will base Wheezy on, so currently we can happily jump to the most
recent ones. There are plans for better interaction with Debian Installer
thanks to generating our udebs properly in linux-2.6 source itself. Also we
are looking forward to using git as tool of maintenance. We d hope that this will
also allow for even better cross distribution collaboration.
Concerning early userspace I plan to release an initramfs-tools with
more generic userspace for the default case and finally also a klibc
only for embedded or tuning cases.
What do you like most in Debian?
For one thing I do like the 2 year release cycle. It is not too long to have
completely outdated software and on the other hand it gives enough time to
really see huge progress from release to release. Also at my institute the
software is is recent enough without too much admin overhead. For servers the
three years support are a bit short, but on the manageable side.
I do enjoy a lot the testing distribution. For my personal use it is very stable
and thus I mainly run testing on my desktop and work boxes. (Occasionally
mixing in things from sid for unbreaking transition or newer security fixes).
Debian is independent and not a commercial entity. I think this is its main
force and even more important these days. I enjoy using the Debian platform a
lot at work thus in return this motivates me to contribute to Debian itself. I also like the
fact that we strive for technical correctness.
Is there some recurrent problem that hinders the progress of Debian?
The New Maintainer process is a strange way to discourage people to
contribute to Debian. It is particularly bureaucratic and a huge waste of time
both for the applicant and his manager. It should be completely thrown
overboard.
One needs a more scalable approach for trust and credibility that also enhances
the technical knowledge for coding and packaging of the applicant.
NM is currently set in stone as any outside critics is automatically rejected.
Young and energetic people are crucial for Debian and the long-term viability of
the project, this is the reason why I d consider the New Maintainer process
as Debian s biggest problem.
Note from Rapha l Hertzog: I must say I do not share this point of view on the New Maintainer process, I have witnessed lots of improvements lately thanks to the addition of the Debian Maintainer status, and to the fact that a good history of contribution can easily subsume the annoying Tasks & Skills questionnaire.
Another thing I miss is professional graphics input both for the desktop
theme and the website. I know that effort has been done there lately and
it is good to see movement there, but the end result is still lacking.
Another trouble of Debian is its marketing capabilities. It should learn to better sell itself. It is the distribution users want to run and use not the rebranded copies of itself with lock-in sugar . Debian is about choice and it offers plenty of it: it is a great default Desktop.
Linus Torvalds doesn t find Debian (and/or Ubuntu) a good platform to hack
on the kernel. Do you know why and what can we do about this?
The Fedora linux-2.6 receives contributions from several Red Hat employed
upstream sub-Maintainers. Thus it typically carries huge patches which are not
yet upstream. As a consequence eventual userland troubles get revealed quite
quickly and are often seen there first. The cutting edge nature of Fedora
rawhide is appealing for many developers.
The usual Debian package division of library development files and the library
itself is traditionally an entry barrier for dev on Debian. Debian got pretty
easily usable these days, although we could and should again improve a lot
more in this sector. Personally I think that Linus hasn t tried Debian
for years.
I have the feeling that the implication of the Debian Kernel team in LKML
has been on the rise. Is that true and how do you explain this?
Ben Hutchings is the Nr.1 contributor for
2.6.33.
He also is top listed as author of patches on stable
2.6.32.
Debian is not listed as organization as many send their linux-2.6 patches from
their corporate or personal email address and thus it won t be attributed to
Debian.
There is currently no means to see how many patches get forwarded for
the stable tree, but I certainly forwarded more then fifty patches.
I was very happy when Greg KH personally thanked me in the
2.6.32.12 release.
In the Squeeze kernel, the firmwares have been stripped and moved into
separate packages in the non-free section. What should a user do to ensure
his system keeps working?
There is a debconf warning on linux-2.6 installation. It is quite clear
that the free linux-2.6 can t depend on the firmware of the non-free
archive (also there is no strict dependency there technically).
On the terminal you d also see warnings by update-initramfs on the
initramfs generation for drivers included in the initramfs.
The debconf warning lists the filename(s) of the missing firmware(s). One can then apt-cache search for the firmware package name and install it via the non-free repository. The check runs against the current loaded modules. The match is not 100% accurate for special cases as the one where the device might be handled well by this driver without firmware, but is accurate enough to warrant the warning.
The set of virtualization technologies that the official Debian kernel
supports seems to change regularly. Which of the currently available
options would you recommend to users who want to build on something
that will last?
KVM has been a smooth ride from day zero. It almost got included instantly
upstream. The uptake it has is great as it sees both dev from Intel and AMD.
Together with libvirt it s management is easy. Also the performance of virtio is
very good.
The linux containers are the thing we are looking forward for enhanced chroots
in the Wheezy schedule. They are also manageable by libvirt.
Xen being the bad outside boy has an incredible shrinking patchset,
thus is fair to expect to see it for Wheezy and beyond. For many it may
come a bit late, but for old hardware without relevant CPU it is there.
Many tend to overstate the importance of the virtualization tech. I d be much
more looking forward to the better Desktop support in newer linux-2.6. The
Desktop is important for linux and something that is in heavy use. The much
better graphics support of the radeon and nouveau drivers:
One comment Liked this article? Click here. My blog is Flattr-enabled.
Using a virtual IP managed by heartbeat is a nice way to work around the slow
fallback behavior with multiple nameservers in /etc/hosts. But unlike ntpd,
bind9 doesn't automatically listen on new IPs on the system.
Here's a cute hack to fix that:
# cat /etc/ha.d/haresources
server01 bind9release IPaddr::10.0.0.3 bind9takeover
# cat /etc/ha.d/resource.d/bind9release
#!/bin/sh
# when giving up resources, reload bind9
case $1 in
stop) /etc/init.d/bind9 reload ;;
esac
exit 0
# cat /etc/ha.d/resource.d/bind9takeover
#!/bin/sh
# on takeover, reload bind9
case $1 in
start) /etc/init.d/bind9 reload ;;
esac
exit 0
A new release 0.9.1 of
Rcpp
went to CRAN and
Debian yesterday.
This version contains mostly bug-fixes and rather few enhancements. The changes are
mostly 'internal fixes' and not user-facing; they mostly address some issues in
memory management which, while not tripping up the unit tests or
common usage, caused trouble for advanced Rcpp modules use and repeated
memory allocation / deallocation. This bit Doug Bates repeatedly, and he put his head down and
debugged the issue. A number of changes fairly deep-down in Rcpp (as well as in R-devel)
later, this looks much better and we owe him a hearfelt Thank You!
A big Thanks! also goes to Luke Tierney who out some new memory debugging
code into R-devel which the --enable-strict-barrier in R 2.13.0 (due in
April) will enable. A related fix is also in R 2.12.2 due on the 25th. The
NEWS entry is below; more detail is in the ChangeLog file in the package.
0.9.1 2011-02-14
o A number of internal changes to the memory allocation / protection of
temporary objects were made---with a heartfelt "Thank You!" to both
Doug Bates for very persistent debugging of Rcpp modules code, and to
Luke Tierney who added additional memory allocation debugging tools
to R-devel (which will be in R 2.13.0 and may also be in R 2.12.2)
o Removed another GNU Make-specific variable from src/Makevars in order
to make the build more portable; this was noticed on FreeBSD
o On *BSD, do not try to compute a stack trace but provide file and
line number (which is the same behaviour as implemented in Windows)
o Fixed an int conversion bug reported by Daniel Sabanes Bove on r-devel,
added unit test as well
o Added unit tests for complex-typed vectors (thanks to Christian Gunning)
o Expanded the Rcpp-quickref vignette (with thanks to Christian Gunning)
o Additional examples were added to the Rcpp-FAQ vignette
Thanks to
CRANberries, there is
also a
diff to the previous release 0.9.0.
As always, even fuller details are on the
Rcpp Changelog page and the
Rcpp page which also
leads to the downloads, the
browseable
doxygen docs and zip files of doxygen output for the standard formats.
A local directory has
source and documentation too.
Questions, comments etc should go to the
rcpp-devel mailing list
off the R-Forge page
Last week, we learned how to get rid of obsolete packages. This time, we re going to learn how to bring back your computer to a state close to a pure Ubuntu/Debian installation.
Thanks to the power of APT, it s easy to add new external repositories and install supplementary software. Unfortunately some of those are not very well maintained. They might contain crappy packages or they might simply not be updated. An external package which was initially working well, can become a burden on system maintenance because it will be interfering with regular updates (for example by requiring a package that should be removed in newer versions of the system).
So my goal for today is to teach you how to identify the packages on your system that are not coming from Debian or Ubuntu. So that you can go through them from time to time and keep only those that you really need. Obsolete packages are a subset of those, but I ll leave them alone. We took care of them last week.
Each (well-formed) APT repository comes with a Release file describing it (example). They provide some values that can be used by APT to identify packages contained in the repository. All official Debian repositories are documented with Origin=Debian (and Origin=Ubuntu for Ubuntu). You can verify the origin value associated to each repository (if any) in the output of apt-cache policy:
[...]
500 http://ftp.debian.org/debian/ lenny/main i386 Packages
release v=5.0.8,o=Debian,a=stable,n=lenny,l=Debian,c=main
origin ftp.debian.org
[...]
From there on, we can simply ask aptitude to compute a list of packages which are both installed and not available in an official Debian repository:
$ aptitude search '?narrow(?installed, !?origin(Debian))!?obsolete' or $ aptitude search '~S ~i !~ODebian !~o'You can replace search with purge or remove if you want to get rid of all the packages listed. But you re more likely to want to remove only a subset of carefully chosen packages you re probably still using some of the software that you installed from external repositories. With synaptic, you can also browse the content of each repository. Click on the Origin button and you have a list of repositories. You can go through the non-Debian repositories and look which packages are installed and up-to-date. But you can do better, you can create a custom view. Click on the menu entry Settings > Filter . Click on New to create a new filter and name it External packages . Unselect everything in the Status tab and keep only Installed .
Go in the Properties tab and here add a new entry Origin Excludes ftp.debian.org . In fact you must replace ftp.debian.org with the hostname of your Debian/Ubuntu mirror. The one that appears on the origin line in the output of apt-cache policy (see the excerpt quoted above in this article).
Note that the term Origin is used to refer to two different things, a field in the release file but also the name of the host for an APT repository. It s a bit confusing if you don t pay attention.
Close the filters window with OK. You now have a new listing of External packages under the Custom Filters screen. You can see which packages are installed and up-to-date and decide whether you really want to keep it. If the package is also provided by Debian/Ubuntu and you want to go back to the version provided by your distribution, you can use the Package > Force version menu entry.
Click here to subscribe to my free newsletter and get my monthly analysis on what s going on in Debian and Ubuntu. Or just follow along via the RSS feed, Identi.ca, Twitter or Facebook.
8 comments Liked this article? Click here. My blog is Flattr-enabled.
C: Heyward's - Johnson C.
C: Rosedale's - downtown
O: Gates - 3205 Main
| Servicio mesa | (21.00) | $8.00 |
| Bebidas | (21.00) | $8.00 |
| Copa salentein | (21.00) | $27.00 |
| Provoleta | (21.00) | $24.00 |
| Asado de tira | (21.00) | $45.00 |
I spent most of the week-end working in the pkg-fonts team SVN
repository and uploading font packages.
The main goal was implementing "Build-Depends:fontforge-nox fontforge"
to make the build of fonts easier and faster when they're built from FontForge source files.
While doing this, I updated a few of these fonts. I also completely
revamped the linux-libertine package which was left pending for quite
some time in our SVN (after initial work by Rog rio Brito). I also definitely adopted the ttf-sinhala-lklug
font package (the only fonts we have for Sinhala language, so far).
Finally, I started going through the list of packages that still
depend on defoma. I prodded maintainers for several of these, the most
outdated ones....to propose adopting the package under the team's
umbrella. We'll see what happens.
Quite productive week-end, indeed (and I even managed to watch the
three 6 nations rugby games, plus running around 30 kilometers...doh).
(The requisite "I have no time any more, it's 10 days late")
It certainly feels like 3 months has passed since I last reflected on her
development. It feels like forever. As usual, we've managed to do a fair bit
in 3 months.
She's doing very well. Growing nicely. Still around the 25th percentile for
weight, but now up to the 50th percentile for height. We just had to upgrade
her from an infant car seat to a "big kid" car seat, because she was maxing
it out on height, not weight. She seems to have handled the transition okay,
but we haven't done any long car trips since making the change.
She's now sitting really well, and pulling herself up into a standing
position. She's cruising around on furniture a bit (and sporting various
bruises to show for it). She started
crawling
at six and a half months, and now there's no stopping her. She's into
everything. No more teeth have appeared yet. Her top teeth have been looking
like they want to do something for ages. I think, given a recent uptick in
fussiness, their appearance may be imminent.
On the food front, she's eating all sorts of things. We've got a few
infant/toddler cookbooks, so we still are predominantly feeding her either a
combination of pureed fruits and vegetables, or a few pureed dishes that
I've prepared (mainly vegetable stews or risotto). She also enjoys bread,
toast and has taken to Vegemite on toast, like a good little Aussie. The
other night we ate out at our local Italian restaurant, and she had some
garlic bread and some penne pasta with marinara sauce. She's definitely
enjoying finger food.
We went back to Australia for a couple of weeks recently, and she handled
the flights extremely well, and had an absolute ball back in Australia.
She'd just started sleeping through the night a few times before we left for
the trip, and didn't sleep as well while we were in Australia, but got back
into the swing of things in under a week of getting back to the US.
She's so much fun at this age. She laughs, she plays. She gets so excited
when I get home from work in the evenings. I love her and being a Dad so
much.
I just need to work more on bilocality. While I did find two geocaches,
one at the South/Noon Train Station (taalverwarringen ), one in the buurt
of the University, I did manage to miss the AW building completely and
utterly. Wow. Except, that Haiku guy came over to talk for a bit (nice).
And I drew. An Atari logo with swirl, for that weird stuff I recently have
been found doing.
More mksh-current news coming soon,
stay tuned. In the meanwhile, I met bonsaikitten IRL (at FOSDEM, yes, too)
who kindly made a live ebuild , i.e. a source package building -current.
Finally let s say a big thank you to the person mostly manning our booth,
gecko2, and to Benny for talking to people, getting That Other Packaging
Thingy working, and pimping the website a bit.
As of Gnash 0.8.8, a workaround was added, calling an NPAPI
function NPN_GetValueForURL to ask the web browser for the
relevant cookies. New diagram:
The downside is that this function was only added in NPAPI version
1.9. This means:
So, the good news is, this problem should almost certainly get
fixed over the coming Debian release cycle, if only because all the
browsers will get updated. In the meantime, disabling cookies (and
removing stale ones from YouTube) is a workaround.
My first ever Linux installation was done in the late
nineties 1998 or thereabouts but was a RedHat 4.5
installation rather than a Debian one. The reason for that was fairly
simple: the Infomagick sixpack CD set that I'd bought contained RedHat,
Debian, and Slackware, but the RedHat installation was the only one that
could be installed directly from CD the other two required me to
write floppies and boot from those to start the installation, and I
wasn't very fond of that idea.
It was only a few years and a few broken RedHat upgrades later that I
saw the light and considered trying out this Debian thing that some of
my classmates were talking about. The fact that I'd just bought my own
computer (rather than having to compete for time with my siblings on my
parents' computer) was a good reason to do a fresh Linux installation.
I'd been planning to install Linux From Scratch, but as what was still
known as the LFS HOWTO told me you'd need a working Linux installation
to do that, I considered my options. Since I'd developed a strong
dislike of RedHat, I wasn't interested in doing another RedHat anymore.
So, I downloaded the most recent version of Debian at the time (Potato
Test Cycle III), wrote it to a CD, and installed. I've probably still got
the CD lying around somewhere.
A few months later, I found these "Linux Gazette" packages in the
archive, with the latest packaged issue being 47, but the latest
upstream version being much higher. Trying to figure out what was going
on, I mailed the maintainer, Adrian Bridgett, who encouraged me to take
over maintenance. Thus began my life of actively contributing to Open Source
software.
In November 2000, I applied to become a Debian Developer. In January
2001, Martin Michlmayr was assigned to be my AM. And in early February
2001, now just over ten years ago, I'd become a Debian Developer. Yes,
that was fast, and no, I probably wasn't really ready yet, at the
time.
Originally, I only cared much about these Linux Gazette packages.
But, as time went on, I started looking around, too. A friend passed me
an old Macintosh Centris 610. As I tried to install Debian on it, I
found that it didn't actually run very well. This turned out to be due
to it having a broken 68LC040 processor, so I bought me another
m68k-based mac, one with a full 68040 processor (a Centris 650). Thus I
became involved in the m68k port and buildd maintenance.
As these old machines came with 80MB or 250MB SCSI hard disks of
which I had none laying around, and a then-recent Debian installation
had minimum requirements of about 200MB, I was in need of network
storage to be able to do anything useful with the mac. NFS didn't work
as expected; the RTC implementation on m68k mac hardware was reverse
engineered and didn't work too well at the time, which meant that the
clock would run slower if the machine was under load, and that in turn
would mean that make would get confused about timestamps, since they
would suddenly appear to originate from the future (in NFS, it's the
server that assigns time stamps, not the client). There was a simple
solution, however; Pavel Macheck had written this neat 'Network Block
Device', which would let the client do its own filesystem on network
storage. Only it wasn't packaged; but then, that was easy to fix. Thus I
started maintaining the single piece of software that I've worked the
most on, to date.
A few years later, Pavel lost interest in maintaining NBD, and handed
over upstream maintenance to me and maintenance of the kernel side
to Paul Clements. Thus began my life in upstream work. And while I
originally joined Debian with the intent of using it as a learning
experience and stepping stone on my way to more "important" free
software, I found that it wasn't as satisfying as was my work
for Debian.
Over the years, there's been this duality where I've felt like I was
doing too much and not enough at the same time. Too much, because the
things I was doing would eat up much of my spare time, leaving little
time left for other hobbies. Not enough, because I witnessed other
people doing much more for Debian than I did, and I wanted to make a
difference.
As the years passed by, many things have changed. Not only in Debian,
but also besides it. I became an independent contractor, focusing mostly
on supporting people in using Debian (although I support them with other
distributions, too). The importance of a port went from 'something which
these weird porter people are doing, and that we should probably help
them with if it doesn't work, but is their problem really' to 'something
that I really really really have to make sure works for my
packages', and back for the port that I cared about most. After
several years of trying, I finally managed to explain to my parents what
this Debian thing is, why it matters, and what my role in the whole
thing is. We did a few releases, some taking longer than I would've
liked. People joined the project, and left again. Some of my friends died. My fame in the
project rose, even though I wasn't aware of it initially; and thus I was
rather surprised when someone asked me whether I was "the
Wouter Verhelst" at a key signing party.
Recently, I've started looking back, and considered the things which
Debian has meant to me. Ten years ago, I was 22, still in college, and
had way too much spare time on my hands. I'd recently gotten my first
Internet installation, and all these online communities were very new to
me. Yes, that was all probably rather late. Debian has changed my life
in many ways; it has allowed me to meet various kinds of people, both
online and in meatspace. I've been to Helsinki, Edinburgh, Mar del
Plata, C ceres, and New York City, places that I might otherwise not
have visited. Each of these trips was an incredible experience that I
have fond memories of; and while the most fond ones originate from
Helsinki, I cherish the memories of each of these trips as some of the
best trips I've ever done. Working on Debian has forced me to learn
about the inner workings of a Linux-based system, which is knowledge
that has helped me tremendously professionally, too. And finally,
working on Debian has given me a unique perspective on this whole FOSS
community, which has helped shape my ethics and my view of the world.
While I believe that I would've subscribed to most of that ideology at
any rate, I'm not sure that the details of my beliefs and understanding
would've been exactly the same. And while I don't agree on every
position that the project subscribes to as a whole, I do believe that
the philosophies that lie at the core of this project contain just the
right mix of pragmatism and ideology that makes it possible for our
project to thrive in a changing world of not only a growing group of
people who subscribe unconditionally to the free software ideals, but
also business people who care mostly about money.
Over the years, somehow I moved from "one of the recent batch of new
Debian Developers" to "someone who's been with the project longer than
most". It still feels weird to see people shut up because you've built
up a reputation in some area, and you give your gut opinion on some
subject without researching it too much. I try not to let that happen
too often.
Today, ten
years and just over a week ago, my life as a Debian Developer
started, and it would change the way I looked at the world, the things I
would do in my spare time, and the people I would meet. I wouldn't have
it any other way.
Thanks, Debian, for what's been a blast; may the next ten years be as
inspiring to me and everyone as the past ten!
There is a social issue with Debian that attacks OSS projects _because_ they use the GPL. Please do not follow these attacks without asking a lawyer.
My problem was: I had a number of databases generated in different machines and I wanted to query them as if they were one, using the database the data came from as a field while querying and while showing results. The databases are SQLite3 files, generated using SQLAlchemy in a Python program.
I solved this by using SQLAlchemy, which was good because I could use the same ORM mapping that the program used. I noticed that the Horizontal Sharding SQLAlchemy extension would fit well the problem, although not perfectly. I had to make some changes in some classes of this extension, and now it works fine.
It was possible to filter the data using database as a criteria, but I couldn't get the database information from each line of a query result. I made a simple patch to SQLAlchemy, which wasn't likely to be introduced in the distribution, but worked for me, and sent it to its bug tracker. The change was included in SQLAlchemy in a very different fashion, as expected, but since I'm using the released version from SQLAlchemy, I kept on using my version of the patch. I don't want to do direct changes in SQLAlchemy source code, so I made the change in my program:
class ShardedSessionShardId(ShardedSession):
def __init__(self, *args, **kwargs):
super(ShardedSessionShardId, self).__init__(*args, **kwargs)
self._query_cls = ShardedQueryShardId
class ShardedQueryShardId(ShardedQuery):
def _execute_and_instances(self, context):
if self._shard_id is not None:
result = self.session.connection(
mapper=self._mapper_zero(),
shard_id=self._shard_id).execute(context.statement, self._params)
news = list(self.instances(result, context))
for new in news:
new.shard_id = self._shard_id
return iter(news)
else:
partial = []
for shard_id in self.query_chooser(self):
result = self.session.connection(
mapper=self._mapper_zero(),
shard_id=shard_id).execute(context.statement, self._params)
news = list(self.instances(result, context))
for new in news:
new.shard_id = shard_id
partial = partial + news
# if some kind of in memory 'sorting'
# were done, this is where it would happen
return iter(partial)
create_session = sessionmaker(class_=ShardedSessionShardId)
class WeakInstanceDictNoIdentity(WeakInstanceDict):
def add(self, state):
# if state.key in self:
# if dict.__getitem__(self, state.key) is not state:
# raise AssertionError("A conflicting state is already "
# "present in the identity map for key %r"
# % (state.key, ))
# else:
dict.__setitem__(self, state.key, state)
self._manage_incoming_state(state)
class ShardedSessionShardId(ShardedSession):
def __init__(self, *args, **kwargs):
super(ShardedSessionShardId, self).__init__(*args, **kwargs)
self._query_cls = ShardedQueryShardId
self._identity_cls = WeakInstanceDictNoIdentity
self.identity_map = self._identity_cls()
class MapperNoIdentity(Mapper):
def _instance_processor(self, context, path, adapter,
polymorphic_from=None, extension=None,
only_load_props=None, refresh_state=None,
polymorphic_discriminator=None):
"""Produce a mapper level row processor callable
which processes rows into mapped instances."""
pk_cols = self.primary_key
if polymorphic_from or refresh_state:
polymorphic_on = None
else:
if polymorphic_discriminator is not None:
polymorphic_on = polymorphic_discriminator
else:
polymorphic_on = self.polymorphic_on
polymorphic_instances = util.PopulateDict(
self._configure_subclass_mapper(
context, path, adapter)
)
version_id_col = self.version_id_col
if adapter:
pk_cols = [adapter.columns[c] for c in pk_cols]
if polymorphic_on is not None:
polymorphic_on = adapter.columns[polymorphic_on]
if version_id_col is not None:
version_id_col = adapter.columns[version_id_col]
identity_class = self._identity_class
def identity_key(row):
return identity_class, tuple([row[column] for column in pk_cols])
new_populators = []
existing_populators = []
load_path = context.query._current_path + path
def populate_state(state, dict_, row, isnew, only_load_props):
if isnew:
if context.propagate_options:
state.load_options = context.propagate_options
if state.load_options:
state.load_path = load_path
if not new_populators:
new_populators[:], existing_populators[:] = \
self._populators(context, path, row,
adapter)
if isnew:
populators = new_populators
else:
populators = existing_populators
if only_load_props:
populators = [p for p in populators
if p[0] in only_load_props]
for key, populator in populators:
populator(state, dict_, row)
session_identity_map = context.session.identity_map
if not extension:
extension = self.extension
translate_row = extension.get('translate_row', None)
create_instance = extension.get('create_instance', None)
populate_instance = extension.get('populate_instance', None)
append_result = extension.get('append_result', None)
populate_existing = context.populate_existing or self.always_refresh
if self.allow_partial_pks:
is_not_primary_key = _none_set.issuperset
else:
is_not_primary_key = _none_set.issubset
def _instance(row, result):
if translate_row:
ret = translate_row(self, context, row)
if ret is not EXT_CONTINUE:
row = ret
if polymorphic_on is not None:
discriminator = row[polymorphic_on]
if discriminator is not None:
_instance = polymorphic_instances[discriminator]
if _instance:
return _instance(row, result)
# determine identity key
if refresh_state:
identitykey = refresh_state.key
if identitykey is None:
# super-rare condition; a refresh is being called
# on a non-instance-key instance; this is meant to only
# occur within a flush()
identitykey = self._identity_key_from_state(refresh_state)
else:
identitykey = identity_key(row)
# instance = session_identity_map.get(identitykey)
# if instance is not None:
# state = attributes.instance_state(instance)
# dict_ = attributes.instance_dict(instance)
# isnew = state.runid != context.runid
# currentload = not isnew
# loaded_instance = False
# if not currentload and \
# version_id_col is not None and \
# context.version_check and \
# self._get_state_attr_by_column(
# state,
# dict_,
# self.version_id_col) != \
# row[version_id_col]:
# raise orm_exc.ConcurrentModificationError(
# "Instance '%s' version of %s does not match %s"
# % (state_str(state),
# self._get_state_attr_by_column(
# state, dict_,
# self.version_id_col),
# row[version_id_col]))
# elif refresh_state:
if refresh_state:
# out of band refresh_state detected (i.e. its not in the
# session.identity_map) honor it anyway. this can happen
# if a _get() occurs within save_obj(), such as
# when eager_defaults is True.
state = refresh_state
instance = state.obj()
dict_ = attributes.instance_dict(instance)
isnew = state.runid != context.runid
currentload = True
loaded_instance = False
else:
# check for non-NULL values in the primary key columns,
# else no entity is returned for the row
if is_not_primary_key(identitykey[1]):
return None
isnew = True
currentload = True
loaded_instance = True
if create_instance:
instance = create_instance(self,
context,
row, self.class_)
if instance is EXT_CONTINUE:
instance = self.class_manager.new_instance()
else:
manager = attributes.manager_of_class(
instance.__class__)
# TODO: if manager is None, raise a friendly error
# about returning instances of unmapped types
manager.setup_instance(instance)
else:
instance = self.class_manager.new_instance()
dict_ = attributes.instance_dict(instance)
state = attributes.instance_state(instance)
state.key = identitykey
# manually adding instance to session. for a complete add,
# session._finalize_loaded() must be called.
state.session_id = context.session.hash_key
session_identity_map.add(state)
if currentload or populate_existing:
if isnew:
state.runid = context.runid
context.progress[state] = dict_
if not populate_instance or \
populate_instance(self, context, row, instance,
only_load_props=only_load_props,
instancekey=identitykey, isnew=isnew) is \
EXT_CONTINUE:
populate_state(state, dict_, row, isnew, only_load_props)
else:
# populate attributes on non-loading instances which have
# been expired
# TODO: apply eager loads to un-lazy loaded collections ?
if state in context.partials or state.unloaded:
if state in context.partials:
isnew = False
(d_, attrs) = context.partials[state]
else:
isnew = True
attrs = state.unloaded
# allow query.instances to commit the subset of attrs
context.partials[state] = (dict_, attrs)
if not populate_instance or \
populate_instance(self, context, row, instance,
only_load_props=attrs,
instancekey=identitykey, isnew=isnew) is \
EXT_CONTINUE:
populate_state(state, dict_, row, isnew, attrs)
if loaded_instance:
state._run_on_load(instance)
if result is not None and \
(not append_result or
append_result(self, context, row, instance,
result, instancekey=identitykey,
isnew=isnew)
is EXT_CONTINUE):
result.append(instance)
return instance
return _instance
_none_set = frozenset([None])
_runid = 1L
_id_lock = util.threading.Lock()
def _new_runid():
global _runid
_id_lock.acquire()
try:
_runid += 1
return _runid
finally:
_id_lock.release()
def query_chooser(query):
try:
return query.shards
except AttributeError:
pass
return tcs.keys()
Sponsoring a package means uploading a package for someone else (usually a new contributor starting out as package maintainer). This is an activity reserved to Debian Developer who are supposed to be knowledgeable about packaging. This article tries to document the process to ensure the sponsor is doing a reasonably good job according to Debian s standards.
Sponsoring a package in the Debian archive is not a trivial matter. It means that you verified the packaging and that it is of the level of quality that Debian strives to have. Let s have a look to what you can and should do when you re sponsoring a package.
Sponsoring the initial upload
Sponsoring a brand new package into Debian requires a thorough review of the Debian packaging. Building the package and testing the software is definitely not enough! You should open every file in the debian directory and look out for potential problems. Here s a checklist that you can use to perform the audit:
apt-get source) and rebuild it (or download the current binary packages with aptitude download). Download the source package to sponsor (usually with dget).
Read the new changelog entry, it should tell you what to expect during the review. The main tool you will use is debdiff, you can run it with two source packages (.dsc files), or two binary packages, or two .changes files (it will then compare all the binary packages listed in the .changes).
If you compare the source packages (excluding upstream files in the case of a new upstream version, for example by filtering the output of debdiff with filterdiff -i '*/debian/*'), you must understand all the changes you see and they should be properly documented in the Debian changelog.
If everything is fine, build the package and compare the binary packages to verify that the changes on the source package have no unexpected consequences (like some files dropped by mistake, missing dependencies, etc.).
You might want to check out the Package Tracking System to verify if the maintainer has not missed something important. Maybe there are translations updates sitting in the BTS that could have been integrated. Maybe the package has been NMUed and the maintainer forgot to integrate the changes from the NMU in his package. Maybe there s a release critical bug that he has left unhandled and that s blocking migration to testing. Whatever. If you find something that she could have done (better), it s time to tell her so that she can improve for next time. And so that she has a better understanding of her responsibilities.
If you have found no problem, upload the new version. Otherwise ask the maintainer to provide you a fixed version.
7 comments Liked this article? Click here. My blog is Flattr-enabled.
LWN Coverage of Eben Moglen's FOSDEM keynote. A very thoughtful article on freedom and how to get there.
While we can, technically, build these free systems that Eben Moglen talks about, history has shown that we won't. People don't care until it's too late until the police is on their doorstep with an arrest warrant (if the regime cares about such details at all.) So it's up to the small minority of people who actually cares and is prepared to pay the price (in money or in being isolated by not being on Facebook or whatever) to create systems that are both Free and also appealing to the uneducated masses. People will move to Free systems only when they're much cheaper and also offer more value as perceived by them, in terms of features, or coolness or whatever. Bling.
We're almost there in terms of software (Firefox managed to get noticeable market share, and people at least talk about OpenOffice / LibreOffice.) We're nowhere close in terms of networks and content, though. Wikipedia and Openstreetmap are the rare exceptions, but there's not much in terms of social networking, searching, messaging, collaboration platforms etc.: all firmly in corporate hands. I'm not sure that Moglen's Freedom Box idea will succeed either: so far, I can't see the must have element for people who don't care about these things that might change the game.
Since a couple of years we ve been handing off security issues of minor orI m happy to confirm, now that it s been announced, that I am that person: point release security co-ordinator. Affected packages If your package fulfils these criteria:
theoretical impact but for which a fix would be desirable at some point, like
certain classes of denial-of-service attacks, off to stable point updates.
We re looking for a person that wants to coordinate this: monitor the Security
Tracker for issues classified as such by the Security Team, converse with
maintainers to get such updates done and coordinate with the stable release
managers on this.
Next.