H ctor Or n Mart nez: Build a Debian package against Debian 8.0 using Download On Demand (DoD) service
In the previous post Open Build Service software architecture has been overviewed. In the current blog post, a tutorial on setting up a package build with OBS from Debian packages is presented.
Steps:
When OBS API database obs-api is created, we need to pick a password for it, provide opensuse . The obs-api package will configure apache2 https webserver (creating a dummy certificate for stretch ) to serve OBS webui.
Add stretch and obs aliases to localhost entry in your /etc/hosts file.
Enable worker by setting ENABLED=1 in /etc/default/obsworker
Try to connect to the web UI https://stretch/
Login into OBS webui, default login credentials: Admin/opensuse).
From command line tool, try to list projects in OBS
If the install proceeds as expected follow to the next step. Ensure all OBS services are running
- Generate a test environment by creating Stretch/SID VM
- Enable experimental repository
- Install OBS server, api, worker and osc CLI packages
- Ensure all OBS services are running
- Create an OBS project for Download on Demand (DoD)
- Create an OBS project linked to DoD
- Adding a package to the project
- Troubleshooting OBS
Install and setup OBS server, api, worker and osc CLI packages# echo "deb http://httpredir.debian.org/debian experimental main" >> /etc/apt/sources.list.d/experimental.list # apt-get update
In the install process mysql database is needed, therefore if mysql server is not setup, a password needs to be provided.# apt-get install obs-server obs-api obs-worker osc
When OBS API database obs-api is created, we need to pick a password for it, provide opensuse . The obs-api package will configure apache2 https webserver (creating a dummy certificate for stretch ) to serve OBS webui.
Add stretch and obs aliases to localhost entry in your /etc/hosts file.
Enable worker by setting ENABLED=1 in /etc/default/obsworker
Try to connect to the web UI https://stretch/
Login into OBS webui, default login credentials: Admin/opensuse).
From command line tool, try to list projects in OBS
Accept dummy certificate and provide credentials (defaults: Admin/opensuse)$ osc -A https://stretch ls
If the install proceeds as expected follow to the next step. Ensure all OBS services are running
Create an OBS project for Download on Demand (DoD) Create a meta project file:# backend services obsrun 813 0.0 0.9 104960 20448 ? Ss 08:33 0:03 /usr/bin/perl -w /usr/lib/obs/server/bs_dodup obsrun 815 0.0 1.5 157512 31940 ? Ss 08:33 0:07 /usr/bin/perl -w /usr/lib/obs/server/bs_repserver obsrun 1295 0.0 1.6 157644 32960 ? S 08:34 0:07 \_ /usr/bin/perl -w /usr/lib/obs/server/bs_repserver obsrun 816 0.0 1.8 167972 38600 ? Ss 08:33 0:08 /usr/bin/perl -w /usr/lib/obs/server/bs_srcserver obsrun 1296 0.0 1.8 168100 38864 ? S 08:34 0:09 \_ /usr/bin/perl -w /usr/lib/obs/server/bs_srcserver memcache 817 0.0 0.6 346964 12872 ? Ssl 08:33 0:11 /usr/bin/memcached -m 64 -p 11211 -u memcache -l 127.0.0.1 obsrun 818 0.1 0.5 78548 11884 ? Ss 08:33 0:41 /usr/bin/perl -w /usr/lib/obs/server/bs_dispatch obsserv+ 819 0.0 0.3 77516 7196 ? Ss 08:33 0:05 /usr/bin/perl -w /usr/lib/obs/server/bs_service mysql 851 0.0 0.0 4284 1324 ? Ss 08:33 0:00 /bin/sh /usr/bin/mysqld_safe mysql 1239 0.2 6.3 1010744 130104 ? Sl 08:33 1:31 \_ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --log-error=/var/log/mysql/error.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/run/mysqld/mysqld.sock --port=3306 # web services root 1452 0.0 0.1 110020 3968 ? Ss 08:34 0:01 /usr/sbin/apache2 -k start root 1454 0.0 0.1 435992 3496 ? Ssl 08:34 0:00 \_ Passenger watchdog root 1460 0.3 0.2 651044 5188 ? Sl 08:34 1:46 \_ Passenger core nobody 1465 0.0 0.1 444572 3312 ? Sl 08:34 0:00 \_ Passenger ust-router www-data 1476 0.0 0.1 855892 2608 ? Sl 08:34 0:09 \_ /usr/sbin/apache2 -k start www-data 1477 0.0 0.1 856068 2880 ? Sl 08:34 0:09 \_ /usr/sbin/apache2 -k start www-data 1761 0.0 4.9 426868 102040 ? Sl 08:34 0:29 delayed_job.0 www-data 1767 0.0 4.8 425624 99888 ? Sl 08:34 0:30 delayed_job.1 www-data 1775 0.0 4.9 426516 101708 ? Sl 08:34 0:28 delayed_job.2 nobody 1788 0.0 5.7 496092 117480 ? Sl 08:34 0:03 Passenger RubyApp: /usr/share/obs/api nobody 1796 0.0 4.9 488888 102176 ? Sl 08:34 0:00 Passenger RubyApp: /usr/share/obs/api www-data 1814 0.0 4.5 282576 92376 ? Sl 08:34 0:22 delayed_job.1000 www-data 1829 0.0 4.4 282684 92228 ? Sl 08:34 0:22 delayed_job.1010 www-data 1841 0.0 4.5 282932 92536 ? Sl 08:34 0:22 delayed_job.1020 www-data 1855 0.0 4.9 427988 101492 ? Sl 08:34 0:29 delayed_job.1030 www-data 1865 0.2 5.0 492500 102964 ? Sl 08:34 1:09 clockworkd.clock www-data 1899 0.0 0.0 87100 1400 ? S 08:34 0:00 /usr/bin/searchd --pidfile --config /usr/share/obs/api/config/production.sphinx.conf www-data 1900 0.1 0.4 161620 8276 ? Sl 08:34 0:51 \_ /usr/bin/searchd --pidfile --config /usr/share/obs/api/config/production.sphinx.conf # OBS worker root 1604 0.0 0.0 28116 1492 ? Ss 08:34 0:00 SCREEN -m -d -c /srv/obs/run/worker/boot/screenrc root 1605 0.0 0.9 75424 18764 pts/0 Ss+ 08:34 0:06 \_ /usr/bin/perl -w ./bs_worker --hardstatus --root /srv/obs/worker/root_1 --statedir /srv/obs/run/worker/1 --id stretch:1 --reposerver http://obs:5252 --jobs 1
$ osc -A https://stretch:443 meta prj Debian:8 -e
<project name= Debian:8 >Visit webUI to check project configuration Create a meta project configuration file:
<title>Debian 8 DoD</title>
<description>Debian 8 DoD</description>
<person userid= Admin role= maintainer />
<repository name= main >
<download arch= x86_64 url= http://deb.debian.org/debian/jessie/main repotype= deb />
<arch>x86_64</arch>
</repository>
</project>
Add the following file, as found at build.opensuse.org$ osc -A https://stretch:443 meta prjconf Debian:8 -e
Visit webUI to check project configuration Create an OBS project linked to DoDRepotype: debian # create initial user Preinstall: base-passwd Preinstall: user-setup # required for preinstall images Preinstall: perl # preinstall essentials + dependencies Preinstall: base-files base-passwd bash bsdutils coreutils dash debconf Preinstall: debianutils diffutils dpkg e2fslibs e2fsprogs findutils gawk Preinstall: gcc-4.9-base grep gzip hostname initscripts insserv libacl1 Preinstall: libattr1 libblkid1 libbz2-1.0 libc-bin libc6 libcomerr2 libdb5.3 Preinstall: libgcc1 liblzma5 libmount1 libncurses5 libpam-modules Preinstall: libpcre3 libsmartcols1 Preinstall: libpam-modules-bin libpam-runtime libpam0g libreadline6 Preinstall: libselinux1 libsemanage-common libsemanage1 libsepol1 libsigsegv2 Preinstall: libslang2 libss2 libtinfo5 libustr-1.0-1 libuuid1 login lsb-base Preinstall: mount multiarch-support ncurses-base ncurses-bin passwd perl-base Preinstall: readline-common sed sensible-utils sysv-rc sysvinit sysvinit-utils Preinstall: tar tzdata util-linux zlib1g Runscripts: base-passwd user-setup base-files gawk VMinstall: libdevmapper1.02.1 Order: user-setup:base-files # Essential packages (this should also pull the dependencies) Support: base-files base-passwd bash bsdutils coreutils dash debianutils Support: diffutils dpkg e2fsprogs findutils grep gzip hostname libc-bin Support: login mount ncurses-base ncurses-bin perl-base sed sysvinit Support: sysvinit-utils tar util-linux # Build-essentials Required: build-essential Prefer: build-essential:make # build script needs fakeroot Support: fakeroot # lintian support would be nice, but breaks too much atm #Support: lintian # helper tools in the chroot Support: less kmod net-tools procps psmisc strace vim # everything below same as for Debian:6.0 (apart from the version macros ofc) # circular dependendencies in openjdk stack Order: openjdk-6-jre-lib:openjdk-6-jre-headless Order: openjdk-6-jre-headless:ca-certificates-java Keep: binutils cpp cracklib file findutils gawk gcc gcc-ada gcc-c++ Keep: gzip libada libstdc++ libunwind Keep: libunwind-devel libzio make mktemp pam-devel pam-modules Keep: patch perl rcs timezone Prefer: cvs libesd0 libfam0 libfam-dev expect Prefer: gawk locales default-jdk Prefer: xorg-x11-libs libpng fam mozilla mozilla-nss xorg-x11-Mesa Prefer: unixODBC libsoup glitz java-1_4_2-sun gnome-panel Prefer: desktop-data-SuSE gnome2-SuSE mono-nunit gecko-sharp2 Prefer: apache2-prefork openmotif-libs ghostscript-mini gtk-sharp Prefer: glib-sharp libzypp-zmd-backend mDNSResponder Prefer: -libgcc-mainline -libstdc++-mainline -gcc-mainline-c++ Prefer: -libgcj-mainline -viewperf -compat -compat-openssl097g Prefer: -zmd -OpenOffice_org -pam-laus -libgcc-tree-ssa -busybox-links Prefer: -crossover-office -libgnutls11-dev # alternative pkg-config implementation Prefer: -pkgconf Prefer: -openrc Prefer: -file-rc Conflict: ghostscript-library:ghostscript-mini Ignore: sysvinit:initscripts Ignore: aaa_base:aaa_skel,suse-release,logrotate,ash,mingetty,distribution-release Ignore: gettext-devel:libgcj,libstdc++-devel Ignore: pwdutils:openslp Ignore: pam-modules:resmgr Ignore: rpm:suse-build-key,build-key Ignore: bind-utils:bind-libs Ignore: alsa:dialog,pciutils Ignore: portmap:syslogd Ignore: fontconfig:freetype2 Ignore: fontconfig-devel:freetype2-devel Ignore: xorg-x11-libs:freetype2 Ignore: xorg-x11:x11-tools,resmgr,xkeyboard-config,xorg-x11-Mesa,libusb,freetype2,libjpeg,libpng Ignore: apache2:logrotate Ignore: arts:alsa,audiofile,resmgr,libogg,libvorbis Ignore: kdelibs3:alsa,arts,pcre,OpenEXR,aspell,cups-libs,mDNSResponder,krb5,libjasper Ignore: kdelibs3-devel:libvorbis-devel Ignore: kdebase3:kdebase3-ksysguardd,OpenEXR,dbus-1,dbus-1-qt,hal,powersave,openslp,libusb Ignore: kdebase3-SuSE:release-notes Ignore: jack:alsa,libsndfile Ignore: libxml2-devel:readline-devel Ignore: gnome-vfs2:gnome-mime-data,desktop-file-utils,cdparanoia,dbus-1,dbus-1-glib,krb5,hal,libsmbclient,fam,file_alteration Ignore: libgda:file_alteration Ignore: gnutls:lzo,libopencdk Ignore: gnutls-devel:lzo-devel,libopencdk-devel Ignore: pango:cairo,glitz,libpixman,libpng Ignore: pango-devel:cairo-devel Ignore: cairo-devel:libpixman-devel Ignore: libgnomeprint:libgnomecups Ignore: libgnomeprintui:libgnomecups Ignore: orbit2:libidl Ignore: orbit2-devel:libidl,libidl-devel,indent Ignore: qt3:libmng Ignore: qt-sql:qt_database_plugin Ignore: gtk2:libpng,libtiff Ignore: libgnomecanvas-devel:glib-devel Ignore: libgnomeui:gnome-icon-theme,shared-mime-info Ignore: scrollkeeper:docbook_4,sgml-skel Ignore: gnome-desktop:libgnomesu,startup-notification Ignore: python-devel:python-tk Ignore: gnome-pilot:gnome-panel Ignore: gnome-panel:control-center2 Ignore: gnome-menus:kdebase3 Ignore: gnome-main-menu:rug Ignore: libbonoboui:gnome-desktop Ignore: postfix:pcre Ignore: docbook_4:iso_ent,sgml-skel,xmlcharent Ignore: control-center2:nautilus,evolution-data-server,gnome-menus,gstreamer-plugins,gstreamer,metacity,mozilla-nspr,mozilla,libxklavier,gnome-desktop,startup-notification Ignore: docbook-xsl-stylesheets:xmlcharent Ignore: liby2util-devel:libstdc++-devel,openssl-devel Ignore: yast2:yast2-ncurses,yast2-theme-SuSELinux,perl-Config-Crontab,yast2-xml,SuSEfirewall2 Ignore: yast2-core:netcat,hwinfo,wireless-tools,sysfsutils Ignore: yast2-core-devel:libxcrypt-devel,hwinfo-devel,blocxx-devel,sysfsutils,libstdc++-devel Ignore: yast2-packagemanager-devel:rpm-devel,curl-devel,openssl-devel Ignore: yast2-devtools:perl-XML-Writer,libxslt,pkgconfig Ignore: yast2-installation:yast2-update,yast2-mouse,yast2-country,yast2-bootloader,yast2-packager,yast2-network,yast2-online-update,yast2-users,release-notes,autoyast2-installation Ignore: yast2-bootloader:bootloader-theme Ignore: yast2-packager:yast2-x11 Ignore: yast2-x11:sax2-libsax-perl Ignore: openslp-devel:openssl-devel Ignore: java-1_4_2-sun:xorg-x11-libs Ignore: java-1_4_2-sun-devel:xorg-x11-libs Ignore: kernel-um:xorg-x11-libs Ignore: tetex:xorg-x11-libs,expat,fontconfig,freetype2,libjpeg,libpng,ghostscript-x11,xaw3d,gd,dialog,ed Ignore: yast2-country:yast2-trans-stats Ignore: susehelp:susehelp_lang,suse_help_viewer Ignore: mailx:smtp_daemon Ignore: cron:smtp_daemon Ignore: hotplug:syslog Ignore: pcmcia:syslog Ignore: avalon-logkit:servlet Ignore: jython:servlet Ignore: ispell:ispell_dictionary,ispell_english_dictionary Ignore: aspell:aspel_dictionary,aspell_dictionary Ignore: smartlink-softmodem:kernel,kernel-nongpl Ignore: OpenOffice_org-de:myspell-german-dictionary Ignore: mediawiki:php-session,php-gettext,php-zlib,php-mysql,mod_php_any Ignore: squirrelmail:mod_php_any,php-session,php-gettext,php-iconv,php-mbstring,php-openssl Ignore: simias:mono(log4net) Ignore: zmd:mono(log4net) Ignore: horde:mod_php_any,php-gettext,php-mcrypt,php-imap,php-pear-log,php-pear,php-session,php Ignore: xerces-j2:xml-commons-apis,xml-commons-resolver Ignore: xdg-menu:desktop-data Ignore: nessus-libraries:nessus-core Ignore: evolution:yelp Ignore: mono-tools:mono(gconf-sharp),mono(glade-sharp),mono(gnome-sharp),mono(gtkhtml-sharp),mono(atk-sharp),mono(gdk-sharp),mono(glib-sharp),mono(gtk-sharp),mono(pango-sharp) Ignore: gecko-sharp2:mono(glib-sharp),mono(gtk-sharp) Ignore: vcdimager:libcdio.so.6,libcdio.so.6(CDIO_6),libiso9660.so.4,libiso9660.so.4(ISO9660_4) Ignore: libcdio:libcddb.so.2 Ignore: gnome-libs:libgnomeui Ignore: nautilus:gnome-themes Ignore: gnome-panel:gnome-themes Ignore: gnome-panel:tomboy Substitute: utempter %ifnarch s390 s390x ppc ia64 Substitute: java2-devel-packages java-1_4_2-sun-devel %else %ifnarch s390x Substitute: java2-devel-packages java-1_4_2-ibm-devel %else Substitute: java2-devel-packages java-1_4_2-ibm-devel xorg-x11-libs-32bit %endif %endif Substitute: yast2-devel-packages docbook-xsl-stylesheets doxygen libxslt perl-XML-Writer popt-devel sgml-skel update-desktop-files yast2 yast2-devtools yast2-packagemanager-devel yast2-perl-bindings yast2-testsuite # # SUSE compat mappings # Substitute: gcc-c++ gcc Substitute: libsigc++2-devel libsigc++-2.0-dev Substitute: glibc-devel-32bit Substitute: pkgconfig pkg-config %ifarch %ix86 Substitute: kernel-binary-packages kernel-default kernel-smp kernel-bigsmp kernel-debug kernel-um kernel-xen kernel-kdump %endif %ifarch ia64 Substitute: kernel-binary-packages kernel-default kernel-debug %endif %ifarch x86_64 Substitute: kernel-binary-packages kernel-default kernel-smp kernel-xen kernel-kdump %endif %ifarch ppc Substitute: kernel-binary-packages kernel-default kernel-kdump kernel-ppc64 kernel-iseries64 %endif %ifarch ppc64 Substitute: kernel-binary-packages kernel-ppc64 kernel-iseries64 %endif %ifarch s390 Substitute: kernel-binary-packages kernel-s390 %endif %ifarch s390x Substitute: kernel-binary-packages kernel-default %endif %define debian_version 800 Macros: %debian_version 800
$ osc -A https://stretch:443 meta prj test -e
<project name= test >Visit webUI to check project configuration Adding a package to the project
<title>test</title>
<description>test</description>
<person userid= Admin role= maintainer />
<repository name= Debian_8.0 >
<path project= Debian:8 repository= main />
<arch>x86_64</arch>
</repository>
</project>
$ osc -A https://stretch:443 co test ; cd test $ mkdir hello ; cd hello ; apt-get source -d hello ; cd - ; $ osc add hello $ osc ci -m "New import" helloThe package should go to dispatched state then get in blocked state while it downloads build dependencies from DoD link, eventually it should start building. Please check the journal logs to check if something went wrong or gets stuck. Visit webUI to check hello package build state OBS logging to the journal Check in the journal logs everything went fine:
$ sudo journalctl -u obsdispatcher.service -u obsdodup.service -u obsscheduler@x86_64.service -u obsworker.service -u obspublisher.serviceTroubleshooting Currently we are facing few issues with web UI:
- Projects are not listed in the general project page (#839137)
- Packages are not listed in the project page
- Statistical Plots are not being drawn in monitor page nor system status in main page
- Text editor does not save modified files (#839133)
- Click on patchinfo/delete in a project page gets 404 (#839143)
- Search function does not work (#839137)
Choice of smart card
For standard PGP use, the
although the software experience will be much the same if lots of individual card readers or USB flash drives are used.
Other devices
One additional idea that comes to mind is a hardware random number generator (TRNG), such as the
Like
In several of my
And here it is zoomed in to just Critical and High:
The line in the middle is the date from which I started the CVE search (2011). The vertical axis is actually linear time, but it s labeled with kernel releases (which are pretty regular). The numerical summary is:

Yesterday
Forgive me, reader, for I have sinned. It has been over a year since my last blog post. Life got busy. Paid work.
Release 0.0.2 of
The next Debian stable release is codenamed Stretch, which I would expect
to be
The nano editor includes nft syntax highlighting.
What are you waiting for to use nftables?
The
Yikes today was Sharad Purnima which means there is about two weeks to go before Diwali and I haven't written anything here all year.
OK new challenge: write 7 substantive blog posts before Diwali. Can I manage to do it? Let's see...
OpenStack Newton is released, and uploaded to Sid
OpenStack Newton was released on the Thursday 6th of October. I was able to upload nearly all of it before the week-end, though there was a bit of hick-ups still, as I forgot to upload python-fixtures 3.0.0 to unstable, and only realized it thanks to some bug reports. As this is a build time dependency, it didn t disrupt Sid users too much, but 38 packages wouldn t build without it. Thanks to Santiago Vila for pointing at the issue here.
As of writing, a lot of the Newton packages didn t migrate to Testing yet. It s been migrating in a very messy way. I d love to improve this process, but I m not sure how, if not filling RC bugs against 250 packages (which would be painful to do), so they would migrate at once. Suggestions welcome.
Bye bye Jenkins
For a few years, I was using Jenkins, together with a post-receive hook to build Debian Stable backports of OpenStack packages. Though nearly a year and a half ago, we had that project to build the packages within the OpenStack infrastructure, and use the CI/CD like OpenStack upstream was doing. This is done, and Jenkins is gone, as of OpenStack Newton.
Current status
As of August, almost all of the packages Git repositories were uploaded to OpenStack Gerrit, and the build now happens in OpenStack infrastructure. We ve been able to build all packages a release OpenStack Newton Debian packages using this system. This non-official jessie backports repository has also been validated using Tempest.
Goodies from Gerrit and upstream CI/CD
It is very nice to have it built this way, so we will be able to maintain a full CI/CD in upstream infrastructure using Newton for the life of Stretch, which means we will have the tools to test security patches virtually forever. Another thing is that now, anyone can propose packaging patches without the need for an Alioth account, by sending a patch for review through Gerrit. It is our hope that this will increase the likeliness of external contribution, for example from 3rd party plugins vendors (ie: networking driver vendors, for example), or upstream contributors themselves. They are already used to Gerrit, and they all expected the packaging to work this way. They are all very much welcome.
The upstream infra: nodepool, zuul and friends
A moment ago,
It's been said that backup is a bit like flossing; everybody knows you should
do it, but nobody does it.
If you want to start flossing, an immediate question is what kind of dental
floss to get and conversely, for backup, which backup software do you want to
rely on? I had some criteria:
It was time for a new work laptop so I got a Dell XPS 15 9950. I wasn't planning
to write a blog post of how to install Debian 8 "Jessie" on the laptop but since
it wasn't just install and use, I will share what is needed to get the wifi and
graphics card to work.
So first download the DVD-1 AMD64 image of Debian 8 from your favorite