Search Results: "gus"

1 June 2020

Paul Wise: FLOSS Activities May 2020

Focus This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Review

Administration
  • nsntrace: talk to upstream about collaborative maintenance
  • Debian: deploy changes, debug issue with GPS markers file generation, migrate bls/DUCK from alioth-archive to salsa
  • Debian website: ran map cron job, synced mirrors
  • Debian wiki: approve accounts, ping folks with bouncing email

Communication

Sponsors The apt-offline work and the libfile-libmagic-perl backports were sponsored. All other work was done on a volunteer basis.

31 May 2020

Enrico Zini: Controversial inventors

Paul-F lix Armand-Delille (3 July 1874 in Fourchambault, Ni vre 4 September 1963) was a physician, bacteriologist, professor, and member of the French Academy of Medicine who accidentally brought about the collapse of rabbit populations throughout much of Europe and beyond in the 1950s by infecting them with myxomatosis.
Charles Franklin Kettering (August 29, 1876 November 25, 1958) sometimes known as Charles "Boss" Kettering[1] was an American inventor, engineer, businessman, and the holder of 186 patents.[2] He was a founder of Delco, and was head of research at General Motors from 1920 to 1947. Among his most widely used automotive developments were the electrical starting motor[3] and leaded gasoline.[4][5] In association with the DuPont Chemical Company, he was also responsible for the invention of Freon refrigerant for refrigeration and air conditioning systems. At DuPont he also was responsible for the development of Duco lacquers and enamels, the first practical colored paints for mass-produced automobiles. While working with the Dayton-Wright Company he developed the "Bug" aerial torpedo, considered the world's first aerial missile.[6] He led the advancement of practical, lightweight two-stroke diesel engines, revolutionizing the locomotive and heavy equipment industries. In 1927, he founded the Kettering Foundation, a non-partisan research foundation. He was featured on the cover of Time magazine on January 9, 1933.
John Charles Cutler (June 29, 1915 February 8, 2003) was a senior surgeon, and the acting chief of the venereal disease program in the United States Public Health Service. After his death, his involvement in several controversial and unethical medical studies of syphilis was revealed, including the Guatemala and the Tuskegee syphilis experiments.
Ivy Ledbetter Lee (July 16, 1877 November 9, 1934) was an American publicity expert and a founder of modern public relations. Lee is best known for his public relations work with the Rockefeller family. His first major client was the Pennsylvania Railroad, followed by numerous major railroads such as the New York Central, the Baltimore and Ohio, and the Harriman lines such as the Union Pacific. He established the Association of Railroad Executives, which included providing public relations services to the industry. Lee advised major industrial corporations, including steel, automobile, tobacco, meat packing, and rubber, as well as public utilities, banks, and even foreign governments. Lee pioneered the use of internal magazines to maintain employee morale, as well as management newsletters, stockholder reports, and news releases to the media. He did a great deal of pro bono work, which he knew was important to his own public image, and during World War I, he became the publicity director for the American Red Cross.[1]

26 May 2020

Russell Coker: Cruises and Covid19

Problems With Cruises GQ has an insightful and detailed article about Covid19 and the Diamond Princess [1], I recommend reading it. FastCompany has a brief article about bookings for cruises in August [2]. There have been many negative comments about this online. The first thing to note is that the cancellation policies on those cruises are more lenient than usual and the prices are lower. So it s not unreasonable for someone to put down a deposit on a half price holiday in the hope that Covid19 goes away (as so many prominent people have been saying it will) in the knowledge that they will get it refunded if things don t work out. Of course if the cruise line goes bankrupt then no-one will get a refund, but I think people are expecting that won t happen. The GQ article highlights some serious problems with the way cruise ships operate. They have staff crammed in to small cabins and the working areas allow transmission of disease. These problems can be alleviated, they could allocate more space to staff quarters and have more capable air conditioning systems to put in more fresh air. During the life of a cruise ship significant changes are often made, replacing engines with newer more efficient models, changing the size of various rooms for entertainment, installing new waterslides, and many other changes are routinely made. Changing the staff only areas to have better ventilation and more separate space (maybe capsule-hotel style cabins with fresh air piped in) would not be a difficult change. It would take some money and some dry-dock time which would be a significant expense for cruise companies. Cruises Are Great People like social environments, they want to have situations where there are as many people as possible without it becoming impossible to move. Cruise ships are carefully designed for the flow of passengers. Both the layout of the ship and the schedule of events are carefully planned to avoid excessive crowds. In terms of meeting the requirement of having as many people as possible in a small area without being unable to move cruise ships are probably ideal. Because there is a large number of people in a restricted space there are economies of scale on a cruise ship that aren t available anywhere else. For example the main items on the menu are made in a production line process, this can only be done when you have hundreds of people sitting down to order at the same time. The same applies to all forms of entertainment on board, they plan the events based on statistical knowledge of what people want to attend. This makes it more economical to run than land based entertainment where people can decide to go elsewhere. On a ship a certain portion of the passengers will see whatever show is presented each night, regardless of whether it s singing, dancing, or magic. One major advantage of cruises is that they are all inclusive. If you are on a regular holiday would you pay to see a singing or dancing show? Probably not, but if it s included then you might as well do it and it will be pretty good. This benefit is really appreciated by people taking kids on holidays, if kids do things like refuse to attend a performance that you were going to see or reject food once it s served then it won t cost any extra. People Who Criticise Cruises For the people who sneer at cruises, do you like going to bars? Do you like going to restaurants? Live music shows? Visiting foreign beaches? A cruise gets you all that and more for a discount price. If Groupon had a deal that gave you a cheap hotel stay with all meals included, free non-alcoholic drinks at bars, day long entertainment for kids at the kids clubs, and two live performances every evening how many of the people who reject cruises would buy it? A typical cruise is just like a Groupon deal for non-stop entertainment from 8AM to 11PM. Will Cruises Restart? The entertainment options that cruises offer are greatly desired by many people. Most cruises are aimed at budget travellers, the price is cheaper than a hotel in a major city. Such cruises greatly depend on economies of scale, if they can t get the ships filled then they would need to raise prices (thus decreasing demand) to try to make a profit. I think that some older cruise ships will be scrapped in the near future and some of the newer ships will be sold to cruise lines that cater to cheap travel (IE P&O may scrap some ships and some of the older Princess ships may be transferred to them). Overall I predict a decrease in the number of middle-class cruise ships. For the expensive cruises (where the cheapest cabins cost over $1000US per person per night) I don t expect any real changes, maybe they will have fewer passengers and higher prices to allow more social distancing or something. I am certain that cruises will start again, but it s too early to predict when. Going on a cruise is about as safe as going to a concert or a major sporting event. No-one is predicting that sporting stadiums will be closed forever or live concerts will be cancelled forever, so really no-one should expect that cruises will be cancelled forever. Whether companies that own ships or stadiums go bankrupt in the mean time is yet to be determined. One thing that s been happening for years is themed cruises. A group can book out an entire ship or part of a ship for a themed cruise. I expect this to become much more popular when cruises start again as it will make it easier to fill ships. In the past it seems that cruise lines let companies book their ships for events but didn t take much of an active role in the process. I think that the management of cruise lines will look to aggressively market themed cruises to anyone who might help, for starters they could reach out to every 80s and 90s pop group those fans are all old enough to be interested in themed cruises and the musicians won t be asking for too much money. Conclusion Humans are social creatures. People want to attend events with many other people. Covid 19 won t be the last pandemic, and it may not even be eradicated in the near future. The possibility of having a society where no-one leaves home unless they are in a hazmat suit has been explored in science fiction, but I don t think that s a plausible scenario for the near future and I don t think that it s something that will be caused by Covid 19.

25 May 2020

Bits from Debian: DebConf20 registration is open!

DebConf20 banner We are happy to announce that registration for DebConf20 is now open. The event will take place from August 23rd to 29th, 2020 at the University of Haifa, in Israel, and will be preceded by DebCamp, from August 16th to 22nd. Although the Covid-19 situation is still rather fluid, as of now, Israel seems to be on top of the situation. Days with less than 10 new diagnosed infections are becoming common and businesses and schools are slowly reopening. As such, we are hoping that, at least as far as regulations go, we will be able to hold an in-person conference. There is more (and up to date) information at the conference's FAQ. Which means, barring a second wave, that there is reason to hope that the conference can go forward. For that, we need your help. We need to know, assuming health regulations permit it, how many people intend to attend. This year probably more than ever before, prompt registration is very important to us. If after months of staying at home you feel that rubbing elbows with fellow Debian Developers is precisely the remedy that will salvage 2020, then we ask that you do register as soon as possible. Sadly, things are still not clear enough for us to make a final commitment to holding an in-person conference, but knowing how many people intend to attend will be a great help in making that decision. The deadline for deciding on postponing, cancelling or changing the format of the conference is June 8th. To register for DebConf20, please visit our website and log into the registration system and fill out the form. You can always edit or cancel your registration, but please note that the last day to confirm or cancel is July 26th, 2020 23:59:59 UTC. We cannot guarantee availability of accommodation, food and swag for unconfirmed registrations. We do suggest that attendees begin making travel arrangements as soon as possible, of course. Please bear in mind that most air carriers allow free cancellations and changes. Any questions about registrations should be addressed to registration@debconf.org. Bursary for travel, accomodation and meals In an effort to widen the diversity of DebConf attendees, the Debian Project allocates a part of the financial resources obtained through sponsorships to pay for bursaries (travel, accommodation, and/or meals) for participants who request this support when they register. As resources are limited, we will examine the requests and decide who will receive the bursaries. They will be destined: Giving a talk, organizing an event or helping during DebConf20 is taken into account when deciding upon your bursary, so please mention them in your bursary application. For more information about bursaries, please visit Applying for a Bursary to DebConf Attention: deadline to apply for bursaries using the registration form before May 31st, 2019 23:59:59 UTC. This deadline is necessary in order to the organisers to have some time to analyze the requests. To register for the Conference, either with or without a bursary request, please visit: https://debconf20.debconf.org/register Participation to DebConf20 is conditional to your respect of our Code of Conduct. We require you to read, understand and abide by this code. DebConf would not be possible without the generous support of all our sponsors, especially our Platinum Sponsor Lenovo and Gold Sponsors deepin and Matanel Foundation. DebConf20 is still accepting sponsors; if you are interested, or think you know of others who would be willing to help, please get in touch!

17 May 2020

Enrico Zini: Art links

Guglielmo Achille Cavellini (11 September 1914 20 November 1990), also known as GAC, was an Italian artist and art collector. After an initial activity as a painter, in the 1940s and 1950s he became one of the major collectors of contemporary Italian abstract art, developing a deep relationship of patronage and friendship with the artists. This experience has its pinnacle in the exhibition Modern painters of the Cavellini collection at the National Gallery of Modern Art in Rome in 1957. In the 1960s Cavellini resumed his activity as an artist, with an ample production spanning from Neo-Dada to performance art to mail art, of which he became one of the prime exponents with the Exhibitions at Home and the Round Trip works. In 1971 he invented autostoricizzazione (self-historicization), upon which he acted to create a deliberate popular history surrounding his existence. He also authored the books Abstract Art (1959), Man painter (1960), Diary of Guglielmo Achille Cavellini (1975), Encounters/Clashes in the Jungle of Art (1977) and Life of a Genius (1989).
Paul Gustave Louis Christophe Dor (/d re /; French: [ ys.tav d . e]; 6 January 1832 23 January 1883[1]) was a French artist, printmaker, illustrator, comics artist, caricaturist, and sculptor who worked primarily with wood-engraving.
Enrico Baj era bravissimo a pij per culo er potere usanno a fantasia. Co quaa sempricit che solo dii granni, raccatta robbe tipo bottoni, pezzi de stoffa, cordoni, passamanerie varie, e l appiccica su a tela insieme aa pittura sua: che pare quasi che sta a gioc ma giocanno giocanno, zitto zitto, riesce a rovesci er monno. >>

1 May 2020

Paul Wise: FLOSS Activities April 2020

Changes

Issues

Review

Administration
  • myrepos: fix the forum
  • Debian: restart non-responsive tor daemon, restart processes due to OOM, apply debian.net changes for DD with expired key
  • Debian wiki: approve accounts
  • Debian QA services: deploy changes, auto-disable oldoldstable pockets

Communication

Sponsors The purple-discord work was sponsored by my employer. All other work was done on a volunteer basis.

29 April 2020

Craig Small: Sending data in a signal

The well-known kill system call has been around for decades and is used to send a signal to another process. The most common use is to terminate or kill another process by sending the KILL or TERM signal but it can be used for a form of IPC, usually around giving the other process a kick to do something. One thing that isn t as well known is besides sending a signal to a process, you can send some data to it. This can either be an integer or a pointer and uses similar semantics to the known kill and signal handler. I came across this when there was a merge request for procps. The main changes are using sigqueue instead of kill in the sender and using a signal action not a signal handler in the receiver. To illustrate this feature, I have a small set of programs called sender and receiver that will pass an integer between them. The Sender The sender program is extremely simple, use a random(ish) from time masked to two bytes, put it in the required union and send the lot to sendqueue.
# include <signal.h>
# include <stdlib.h>
# include <stdio.h>
# include <time.h>
int main(int argc, char **argv)
 
    union sigval sigval;
    pid_t pid;
    if (argc < 2   (pid = atoi(argv[1])) < 0)
	return EXIT_FAILURE;
    sigval.sival_int = time(NULL) &amp; 0xff;
    printf("sender: sending %d to PID %d\n",
        sigval.sival_int, pid);
    sigqueue(pid, SIGUSR1, sigval);
    return EXIT_SUCCESS;
 
The key lines are 13 and 16 where the random (ish) integer is stored in the sigval union and then sent to the other process with the sigqueue. The receiver The receiver just sets up the signal handler, sends its PID (so I know what to tell the sender) and sits in a sleeping loop.
# include <stdio.h>
# include <stdlib.h>
# include <sys/types.h>
# include <unistd.h>
# include <signal.h>
void signal_handler(int signum, siginfo_t *siginfo, void *ucontext)
 
    if (signum != SIGUSR1) return;
    if (siginfo->si_code != SI_QUEUE) return;
    printf("receiver: Got value %d\n",
	    siginfo->si_int);
 
int main(int argc, char **argv)
 
    pid_t pid = getpid();
    struct sigaction signal_action;
    printf("receiver: PID is %d\n", pid);
    signal_action.sa_sigaction = signal_handler;
    sigemptyset (&amp;signal_action.sa_mask);
    signal_action.sa_flags = SA_SIGINFO;
    sigaction(SIGUSR1, &amp;signal_action, NULL);
    while(1) sleep(100);
    return EXIT_SUCCESS;
 
Lines 16-26 setup the signal handler. The main difference here is SA_SIGINFO used for the signal flags and sigaction references a sa_sigaction function rather than sa_handler. We need to use a different function because the sigaction only is passed the signal number but we need more information, including the integer that the sender process stored in sigval. Lines 7-14 are the signal handler function itself. It first checks that the receiver process got the correct signal (SIGUSR1 in this case) and that we got this signal from sigqueue because the type is SI_QUEUE. Checking the type of signal is important because different signals give you different data. For example, if you signalled this process with kill then si_int is undefined. The result As a proof of concept, the results are not terribly exciting. We see the sender say what it will be sending and the receiver saying it got it. It was useful to get some results, especially when things went wrong.
$ ./receiver &amp;
[1] 370216
receiver: PID is 370216
$ ./sender 370216
sender: sending 133 to (gdPID 370216
receiver: Got value 133
Gotchas While testing the two process there was two gotchas I encountered. GDB and the siginfo structure The sigaction manual page shows a simple siginfo_t structure, however when looking at what is passed to the signal handler, it s much more complicated.
(gdb) p *siginfo
$2 =  si_signo = 10, si_errno = 0, si_code = -1, __pad0 = 0, _sifields =  _pad =  371539, 1000, 11, 32766, 0 <repeats 24 times> , _kill =  si_pid = 371539, si_uid = 1000 , _timer =  si_tid = 371539, 
      si_overrun = 1000, si_sigval =  sival_int = 11, sival_ptr = 0x7ffe0000000b , _rt =  si_pid = 371539, si_uid = 1000, si_sigval =  sival_int = 11, sival_ptr = 0x7ffe0000000b , _sigchld =  
      si_pid = 371539, si_uid = 1000, si_status = 11, si_utime = 0, si_stime = 0 , _sigfault =  si_addr = 0x3e80005ab53, si_addr_lsb = 11, _bounds =  _addr_bnd =  _lower = 0x0, _upper = 0x0 , _pkey = 0 , 
    _sigpoll =  si_band = 4294967667539, si_fd = 11 , _sigsys =  _call_addr = 0x3e80005ab53, _syscall = 11, _arch = 32766 
(gdb) p siginfo->_sifields._rt.si_sigval.sival_int
$3 = 11
So the integer is stored in a union in a structure in a structure. Much harder to find than just simply sival_int. The pointer is just a pointer So perhaps sending an integer is not enough. The sigval is a union with an integer and a pointer. Could a string be sent instead? I changed line 13 of the sender so it used a string instead of an integer.
    sigval.sival_ptr = "Hello, World!"
The receiver needed a minor adjustment to print out the string. I tried this and the receiver segmentation faulted. What was going on? The issue is the set of system calls does a simple passing of the values. So if the sender sends a pointer to a string located at 0x1234567 then the receiver will have a pointer to the same location. When the receiver tries to dereference the sival_ptr, it is pointing to memory that is not owned by it but by another process (the sender) so it segmentation faults. The solution would be to use shared memory between the processes. The signal queue would then use the pointer to the shared memory and, in theory, all would be well.

26 April 2020

Enrico Zini: Some Italian women

Artemisia Gentileschi - Wikipedia
art history people archive.org
Artemisia Lomi or Artemisia Gentileschi (US: / d nt l ski, -ti -/, Italian: [arte mi zja d enti leski]; July 8, 1593 c. 1656) was an Italian Baroque painter, now considered one of the most accomplished seventeenth-century artists working in the dramatic style of Caravaggio. In an era when women had few opportunities to pursue artistic training or work as professional artists, Artemisia was the first woman to become a member of the Accademia di Arte del Disegno in Florence and had an international clientele.
Maria Pellegrina Amoretti (1756 1787), was an Italian lawyer. She is referred to as the first woman to graduate in law in Italy, and the third woman to earn a degree.
Laura Maria Caterina Bassi (October 1711 20 February 1778) was an Italian physicist and academic. She received a doctoral degree in Philosophy from the University of Bologna in May 1732. She was the first woman to earn a professorship in physics at a university. She is recognized as the first woman in the world to be appointed a university chair in a scientific field of studies. Bassi contributed immensely to the field of science while also helping to spread the study of Newtonian mechanics through Italy.
Maria Gaetana Agnesi (UK: / n je zi/ an-YAY-zee,[1] US: / n -/ ahn-,[2][3] Italian: [ma ri a ae ta na a zi, - e z-];[4] 16 May 1718 9 January 1799) was an Italian mathematician, philosopher, theologian, and humanitarian. She was the first woman to write a mathematics handbook and the first woman appointed as a mathematics professor at a university.[5]
Elena Lucrezia Cornaro Piscopia (US: /k r n ro p sko pi /,[4] Italian: [ lena lu kr ttsja kor na ro pi sk pja]) or Elena Lucrezia Corner (Italian: [kor n r]; 5 June 1646 26 July 1684), also known in English as Helen Cornaro, was a Venetian philosopher of noble descent who in 1678 became one of the first women to receive an academic degree from a university, and the first to receive a Doctor of Philosophy degree.
Maria Tecla Artemisia Montessori (/ m nt s ri/ MON-tiss-OR-ee, Italian: [ma ri a montes s ri]; August 31, 1870 May 6, 1952) was an Italian physician and educator best known for the philosophy of education that bears her name, and her writing on scientific pedagogy. At an early age, Montessori broke gender barriers and expectations when she enrolled in classes at an all-boys technical school, with hopes of becoming an engineer. She soon had a change of heart and began medical school at the Sapienza University of Rome, where she graduated with honors in 1896. Her educational method is still in use today in many public and private schools throughout the world.
Rita Levi-Montalcini OMRI OMCA (US: / le vi mo nt l t i ni, l v-, li vi m nt l -/, Italian: [ ri ta l vi montal t i ni]; 22 April 1909 30 December 2012) was an Italian Nobel laureate, honored for her work in neurobiology. She was awarded the 1986 Nobel Prize in Physiology or Medicine jointly with colleague Stanley Cohen for the discovery of nerve growth factor (NGF). From 2001 until her death, she also served in the Italian Senate as a Senator for Life. This honor was given due to her significant scientific contributions. On 22 April 2009, she became the first Nobel laureate ever to reach the age of 100, and the event was feted with a party at Rome's City Hall. At the time of her death, she was the oldest living Nobel laureate.
Margherita Hack Knight Grand Cross OMRI (Italian: [mar e ri ta (h)ak]; 12 June 1922 29 June 2013) was an Italian astrophysicist and scientific disseminator. The asteroid 8558 Hack, discovered in 1995, was named in her honour.
Samantha Cristoforetti (Italian pronunciation: [sa manta kristofo retti]; born 26 April 1977, in Milan) is an Italian European Space Agency astronaut, former Italian Air Force pilot and engineer. She holds the record for the longest uninterrupted spaceflight by a European astronaut (199 days, 16 hours), and until June 2017 held the record for the longest single space flight by a woman until this was broken by Peggy Whitson and later by Christina Koch. She is also the first Italian woman in space. Samantha Cristoforetti is also known as the first person who brewed an espresso in space.

14 April 2020

Steve Kemp: A collection of simple golang tools

In a previous job I wrote some simple utilities which were helpful for continuous-integration pipelines. Most of these were very very simple, for example: Each of these simple tools were run of the mill shell-scripts, or simple binaries. I've been putting together a simple deployable tool which lets you run them easily, so here it is: The idea is that you install the tool, then run the commands by specify the appropriate sub-command:
sysbox validate-yaml [path/to/find/files/beneath]
Of course if you symlink validate-yaml to sysbox you don't need to prefix with the name - so this is just like busybox in this regard. Might be interesting to some.

1 April 2020

Russ Allbery: Review: A Grand and Bold Thing

Review: A Grand and Bold Thing, by Ann Finkbeiner
Publisher: Free Press
Copyright: August 2010
ISBN: 1-4391-9647-8
Format: Kindle
Pages: 200
With the (somewhat excessively long) subtitle of An Extraordinary New Map of the Universe Ushering In a New Era of Discovery, this is a history of the Sloan Digital Sky Survey. It's structured as a mostly chronological history of the project with background profiles on key project members, particularly James Gunn. Those who follow my blog will know that I recently started a new job at Vera C. Rubin Observatory (formerly the Large Synoptic Survey Telescope). Our goal is to take a complete survey of the night sky several times a week for ten years. That project is the direct successor of the Sloan Digital Sky Survey, and its project team includes many people who formerly worked on Sloan. This book (and another one, Giant Telescopes) was recommended to me as a way to come up to speed on the history of this branch of astronomy. Before reading this book, I hadn't understood how deeply the ready availability of the Sloan sky survey data had changed astronomy. Prior to the availability of that survey data, astronomers would develop theories and then try to book telescope time to make observations to test those theories. That telescope time was precious and in high demand, so was not readily available, and was vulnerable to poor weather conditions (like overcast skies) once the allocated time finally arrived. The Sloan project changed all of that. Its output was a comprehensive sky survey available digitally whenever and wherever an astronomer needed it. One could develop a theory and then search the Sloan Digital Sky Survey for relevant data and, for at least some types of theories, test that theory against the data without needing precious telescope time or new observations. It was a transformational change in astronomy, made possible by the radical decision, early in the project, to release all of the data instead of keeping it private to a specific research project. The shape of that change is one takeaway from this book. The other is how many problems the project ran into trying to achieve that goal. About a third of the way into this book, I started wondering if the project was cursed. So many things went wrong, from institutional politics through equipment failures to software bugs and manufacturing problems with the telescope mirror. That makes it all the more impressive how much impact the project eventually had. It's also remarkable just how many bad things can happen to a telescope mirror without making the telescope unusable. Finkbeiner provides the most relevant astronomical background as she tells the story so that the unfamiliar reader can get an idea of what questions the Sloan survey originally set out to answer (particularly about quasars), but this is more of a project history than a popular astronomy book. There's enough astronomy here for context, but not enough to satisfy curiosity. If you're like me, expect to have your curiosity piqued, possibly resulting in buying popular surveys of current astronomy research. (At least one review is coming soon.) Obviously this book is of special interest to me because of my new field of work, my background at a research university, and because it features some of my co-workers. I'm not sure how interesting it will be to someone without that background and personal connection. But if you've ever been adjacent to or curious about how large-scale science projects are done, this is a fascinating story. Both the failures and problems and the way they were eventually solved is different than how the more common stories of successful or failed companies are told. (It helps, at least for me, that the shared goal was to do science, rather than to make money for a corporation whose fortunes are loosely connected to those of the people doing the work.) Recommended if this topic sounds at all interesting. Rating: 7 out of 10

24 March 2020

Anisa Kuci: Outreachy post 5 - Final report

This is my last Outreachy blogpost, as my internship unfortunately has come to an end. It was a blast! Through the Outreachy internship I gained a lot of valuable knowledge and even though I dont know what the future will bring, I am more confident about myself, my skills in fundraising and my technical abilities now. During the contribution phase I did quite a lot of research so I could come up with relevant information to add on the Debian wiki as part of the contribution phase tasks. This was helpful for me to build a more deep understanding of the Debian project, the DebConf conferences and the general style of working within Debian. During this phase I completed most of the tasks given and got onto the mailing lists and IRC channels which are public. This was quite an intense experience by itself as it was like digging into a new job but in a competitive situation as other applicants, of course, were also putting in their best to get one of the 50 Outreachy internships (two in Debian) that were available. When I got selected I also got access to the sponsors@debconf.org email address and the private git repos as they would be needed for me to work on fundraising. I was also provided with an @debconf.org email address that is the one I use for sponsors communication. I learned what makes communication look professional and how to think of the recipient when formulating emails, sending replies or creating marketing messages. As the internship continued I started to learn how the DebConf organizing structure works, with a particular attention on the fundraising team. I have also quickly been given the responsibility to reach out to 100 potential sponsors and combine learning and working experience very nicely. I was given responsibility of working on the fundraising material (flyer and brochure) for DebConf20, so using LaTeX I updated the files in order to remove the translation system, I added the Israeli currency and the visual design chosen from a public logo proposal contest that is held for each DebConf. My creative side could also be put to good use as the team trusted me with selecting the images for the brochure and making sure that the usage rights granted and attribution are all compliant to Free Software/Creative Commons licences. I have continuously maintained those materials and was very happy that long term team members even asked me to approve changes they proposed on the fundraising material. The MiniDebConf team for Regensburg have taken the files I created and made them into their fundraising material (brochure, flyer) which is great to see. MiniDebConf Regensburg commits I have profited so much from other people s work, so I am very happy if community members can build on mine. Afterall sharing is caring! Through the Outreachy travel stipend I was able to attend FOSDEM 2020 and besides meeting many friends from the community and having a fundraising meeting with a sponsor, I was able to also support distributing the DebConf20 fundraising material both in electronic form and as printed versions. During the Outreachy internship I did two communication waves to potential sponsors and maintained the daily communication with them. I followed the documentation that was already available in the Debian wiki guiding me through the process of invoicing sponsors through the organizations that Debian collaborates with. As I have been in frequent contact with the sponsors, I have continuously updated the DebConf20 website with sponsor logos and links through git files, improving web pages on the wafer system and doing 18 merge requests in total. DebConf20 website merge requests I completed six email templates for the DebConf sponsor communication campaigns. While two of them have already been very useful, I hope my other proposals will be the ones used for the next waves of sponsors contacting. I committed them to the sponsors git repo so they can be easily accessible and well documented for other DebConf teams or can be recycled for smaller Debian events. Unfortunately, due to the insecurity about travel around the Corona virus pandemic, the third wave of communication, originally scheduled for the end of my internship in March, has been held. I have committed to the team to do this once the travel restrictions have been lifted and we know that DebConf20 can proceed as planned for August 2020. So yeah, I am definitely going to stay around and continue to help with DebConf fundraising! DebConf has been held for two decades and it also has a generic website, so I also updated the static sponsors page on the main website as it is the first landing page for everyone especially for people who are not necessarily insiders in Debian or DebConf. I have completed a Python tutorial, got the general understanding of the programming language and made some smaller contributions to improve existing tools in the fundraising team based on the learning. The internship was also focused on documentation, so during the whole internship I have kept notes in order to be able to improve existing documentation and I have also written new material, especially on the tasks that I have been working on more closely: Every DebConf has a new team every year, in a new country, so I hope this documentation I have worked on will be useful as a jump-starter for them to organize and finance their events. I would like to take a moment to thank again my mentors Daniel and Karina for all their support, it has been great working with them. Having them as an example, I have learned a lot. Also warm thanks to the DebConf global and local teams which have been very welcoming and always supportive during my internship. So, as you might already know, if you have read my other Outreachy blog posts, this has been a great experience for me! Outreachy provides an awesome opportunity that would not be available if it was not for the generous sponsors and volunteers making such a program for people from underrepresented groups in the FLOSS community. I really encourage people from these groups to find the confidence within themselves and apply for Outreachy! I love Debian

22 March 2020

Sylvestre Ledru: Some clang rebuild results (8.0.1, 9.0.1 & 10rc2)

As part of the LLVM release cycle, I am continuing rebuilding the Debian archive with clang instead of gcc to evaluate potential regressions. Processed results are available on the website: https://clang.debian.net/status.php - Now includes some fancy graphs to show the evolution
Raw logs are published on github: https://github.com/opencollab/clang.debian.net/tree/master/logs Since my last blog post on the subject (August 2017), Clang is more and more present in the tech ecosystem. It is now the compiler used to build Firefox and Chrome upstream binaries on all the supported architectures/operating systems. More architectures are supported, it has a new linker (lld), a new hybrid IR (MLIR), a lot of checkers in clang-tidy, cross-language linking with Rust, etc.

Results
Now, about Debian results, we rebuilt using 8.0.1, 9.0.1 and 10.0rc2. Results are pretty similar to what we had with previous versions: between 4 to 5% of packages are failing when gcc is replaced by clang.
Some clang rebuild results (8.0.1, 9.0.1 &amp; 10rc2)

Even if most of the software are still using gcc as compiler, we can see that clang has a positive effect on code quality. With many different kinds of errors and warnings found clang over the years, we noticed a steady decline of the number of errors. For example, the number of incorrect C/C++ main declarations has been decreasing years after years:
Some clang rebuild results (8.0.1, 9.0.1 &amp; 10rc2)

Errors found
The biggest offender is still the qmake changes which doesn't allow the used workaround (replacing /usr/bin/gcc by /usr/bin/clang) - about 250 errors. Most of these packages would probably compile fine with clang. More on the Qt bug tracker. The workaround proposed in the bug isn't applicable for us as we use the dropped-in replacement of the
The second error is still some differences in symbol generation. Unlike gcc, it seems that clang doesn't generate some symbols (or adds some). As a bunch of Debian packages are checking the list of symbols in the library (for ABI management), the build fails on purpose. For example, with libcec, the symbol _ZN10P8PLATFORM14CConditionImplD1Ev@Base 3.1.0 isn't generated anymore. I am not expecting this to be a big deal: the generated libraries probably works most of the time. More on C++ symbol management in Debian.

Current status
As previously said in a blog post, I don't think there is a strong intensive to go away from gcc for most of the Linux distributions. The big reason for BSD was the license (even if the move to the Apache 2 license wasn't received positively by some of them).
While the LLVM/clang ecosystem clearly won the tooling battle, as a C/C++ compiler, gcc is still an excellent compiler which supports more architecture and more languages.
In term of new warnings and checks, as the clang community moved the efforts in clang-tidy (which requires more complex tooling), out of the box, gcc provides a better experience (as example, see the Firefox meta bug to build with -Werror with the default warnings using gcc 9, gcc 10 and clang trunk for example).

Next steps
I see some potential next steps to decrease the number of failure:
  • Workaround the Qt/Qmake issue
  • Fix the objective-c header include issues (echo "#include <objc/objc.h>" > foo.m && clang -c foo.m is currently failing)
  • Identify why clang generates more/less symbols that gcc in the library and try to fix that
  • Rebuild the archive with clang-7 - Seems that I have some data problem

Many thanks to Lucas Nussbaum for the rebuilds.

Sylvestre Ledru: Some clang rebuild results (8.0.1, 9.0.1 & 10rc2)

As part of the LLVM release cycle, I am continuing rebuilding the Debian archive with clang instead of gcc to evaluate potential regressions. Processed results are available on the website: https://clang.debian.net/status.php - Now includes some fancy graphs to show the evolution
Raw logs are published on github: https://github.com/opencollab/clang.debian.net/tree/master/logs Since my last blog post on the subject (August 2017), Clang is more and more present in the tech ecosystem. It is now the compiler used to build Firefox and Chrome upstream binaries on all the supported architectures/operating systems. More architectures are supported, it has a new linker (lld), a new hybrid IR (MLIR), a lot of checkers in clang-tidy, cross-language linking with Rust, etc.

Results
Now, about Debian results, we rebuilt using 8.0.1, 9.0.1 and 10.0rc2. Results are pretty similar to what we had with previous versions: between 4 to 5% of packages are failing when gcc is replaced by clang.
Some clang rebuild results (8.0.1, 9.0.1 &amp; 10rc2)

Even if most of the software are still using gcc as compiler, we can see that clang has a positive effect on code quality. With many different kinds of errors and warnings found clang over the years, we noticed a steady decline of the number of errors. For example, the number of incorrect C/C++ main declarations has been decreasing years after years:
Some clang rebuild results (8.0.1, 9.0.1 &amp; 10rc2)

Errors found
The biggest offender is still the qmake changes which doesn't allow the used workaround (replacing /usr/bin/gcc by /usr/bin/clang) - about 250 errors. Most of these packages would probably compile fine with clang. More on the Qt bug tracker. The workaround proposed in the bug isn't applicable for us as we use the dropped-in replacement of the compiler.
The second error is still some differences in symbol generation. Unlike gcc, it seems that clang doesn't generate some symbols (or adds some). As a bunch of Debian packages are checking the list of symbols in the library (for ABI management), the build fails on purpose. For example, with libcec, the symbol _ZN10P8PLATFORM14CConditionImplD1Ev@Base 3.1.0 isn't generated anymore. I am not expecting this to be a big deal: the generated libraries probably works most of the time. More on C++ symbol management in Debian.
I reported this bug upstream a while back: https://bugs.llvm.org/show_bug.cgi?id=30441

Current status
As previously said in a blog post, I don't think there is a strong intensive to go away from gcc for most of the Linux distributions. The big reason for BSD was the license (even if the move to the Apache 2 license wasn't received positively by some of them).
While the LLVM/clang ecosystem clearly won the tooling battle, as a C/C++ compiler, gcc is still an excellent compiler which supports more architecture and more languages.
In term of new warnings and checks, as the clang community moved the efforts in clang-tidy (which requires more complex tooling), out of the box, gcc provides a better experience (as example, see the Firefox meta bug to build with -Werror with the default warnings using gcc 9, gcc 10 and clang trunk for example).

Next steps
I see some potential next steps to decrease the number of failure:
  • Workaround the Qt/Qmake issue
  • Fix the objective-c header include issues (echo "#include <objc/objc.h>" > foo.m && clang -c foo.m is currently failing)
  • Identify why clang generates more/less symbols that gcc in the library and try to fix that
  • Rebuild the archive with clang-7 - Seems that I have some data problem

Many thanks to Lucas Nussbaum for the rebuilds.

17 November 2017

Jonathan Carter: I am now a Debian Developer

It finally happened On the 6th of April 2017, I finally took the plunge and applied for Debian Developer status. On 1 August, during DebConf in Montr al, my application was approved. If you re paying attention to the dates you might notice that that was nearly 4 months ago already. I was trying to write a story about how it came to be, but it ended up long. Really long (current draft is around 20 times longer than this entire post). So I decided I d rather do a proper bio page one day and just do a super short version for now so that someone might end up actually reading it. How it started In 1999 no wait, I can t start there, as much as I want to, this is a short post, so In 2003, I started doing some contract work for the Shuttleworth Foundation. I was interested in collaborating with them on tuXlabs, a project to get Linux computers into schools. For the few months before that, I was mostly using SuSE Linux. The open source team at the Shuttleworth Foundation all used Debian though, which seemed like a bizarre choice to me since everything in Debian was really old and its boot-floppies installer program kept crashing on my very vanilla computers.

SLUG (Schools Linux Users Group) group photo. SLUG was founded to support the tuXlab schools that ran Linux.

My contract work then later turned into a full-time job there. This was a big deal for me, because I didn t want to support Windows ever again, and I didn t ever think that it would even be possible for me to get a job where I could work on free software full time. Since everyone in my team used Debian, I thought that I should probably give it another try. I did, and I hated it. One morning I went to talk to my manager, Thomas Black, and told him that I just don t get it and I need some help. Thomas was a big mentor to me during this phase. He told me that I should try upgrading to testing, which I did, and somehow I ended up on unstable, and I loved it. Before that I used to subscribe to a website called freshmeat that listed new releases of upstream software and then, I would download and compile it myself so that I always had the newest versions of everything. Debian unstable made that whole process obsolete, and I became a huge fan of it. Early on I also hit a problem where two packages tried to install the same file, and I was delighted to find how easily I could find package state and maintainer scripts and fix them to get my system going again. Thomas told me that anyone could become a Debian Developer and maintain packages in Debian and that I should check it out and joked that maybe I could eventually snap up highvoltage@debian.org . I just laughed because back then you might as well have told me that I could run for president of the United States, it really felt like something rather far-fetched and unobtainable at that point, but the seed was planted :) Ubuntu and beyond

Ubuntu 4.10 default desktop Image from distrowatch

One day, Thomas told me that Mark is planning to provide official support for Debian unstable. The details were sparse, but this was still exciting news. A few months later Thomas gave me a CD with just warty written on it and said that I should install it on a server so that we can try it out. It was great, it used the new debian-installer and installed fine everywhere I tried it, and the software was nice and fresh. Later Thomas told me that this system is going to be called Ubuntu and the desktop edition has naked people on it. I wasn t sure what he meant and was kind of dumbfounded so I just laughed and said something like Uh ok . At least it made a lot more sense when I finally saw the desktop pre-release version and when it got the byline Linux for Human Beings . Fun fact, one of my first jobs at the foundation was to register the ubuntu.com domain name. Unfortunately I found it was already owned by a domain squatter and it was eventually handled by legal. Closer to Ubuntu s first release, Mark brought over a whole bunch of Debian developers that was working on Ubuntu over to the foundation and they were around for a few days getting some sun. Thomas kept saying Go talk to them! Go talk to them! , but I felt so intimidated by them that I couldn t even bring myself to walk up and say hello. In the interest of keeping this short, I m leaving out a lot of history but later on, I read through the Debian packaging policy and really started getting into packaging and also discovered Daniel Holbach s packaging tutorials on YouTube. These helped me tremendously. Some day (hopefully soon), I d like to do a similar video series that might help a new generation of packagers. I ve also been following DebConf online since DebConf 7, which was incredibly educational for me. Little did I know that just 5 years later I would even attend one, and another 5 years after that I d end up being on the DebConf Committee and have also already been on a local team for one.

DebConf16 Organisers, Photo by Jurie Senekal.

It s been a long journey for me and I would like to help anyone who is also interested in becoming a Debian maintainer or developer. If you ever need help with your package, upload it to https://mentors.debian.net and if I have some spare time I ll certainly help you out and sponsor an upload. Thanks to everyone who have helped me along the way, I really appreciate it!

2 November 2017

Antoine Beaupr : October 2017 report: LTS, feed2exec beta, pandoc filters, git mediawiki

Debian Long Term Support (LTS) This is my monthly Debian LTS report. This time I worked on the famous KRACK attack, git-annex, golang and the continuous stream of GraphicsMagick security issues.

WPA & KRACK update I spent most of my time this month on the Linux WPA code, to backport it to the old (~2012) wpa_supplicant release. I first published a patchset based on the patches shipped after the embargo for the oldstable/jessie release. After feedback from the list, I also built packages for i386 and ARM. I have also reviewed the WPA protocol to make sure I understood the implications of the changes required to backport the patches. For example, I removed the patches touching the WNM sleep mode code as that was introduced only in the 2.0 release. Chunks of code regarding state tracking were also not backported as they are part of the state tracking code introduced later, in 3ff3323. Finally, I still have concerns about the nonce setup in patch #5. In the last chunk, you'll notice peer->tk is reset, to_set to negotiate a new TK. The other approach I considered was to backport 1380fcbd9f ("TDLS: Do not modify RNonce for an TPK M1 frame with same INonce") but I figured I would play it safe and not introduce further variations. I should note that I share Matthew Green's observations regarding the opacity of the protocol. Normally, network protocols are freely available and security researchers like me can easily review them. In this case, I would have needed to read the opaque 802.11i-2004 pdf which is behind a TOS wall at the IEEE. I ended up reading up on the IEEE_802.11i-2004 Wikipedia article which gives a simpler view of the protocol. But it's a real problem to see such critical protocols developed behind closed doors like this. At Guido's suggestion, I sent the final patch upstream explaining the concerns I had with the patch. I have not, at the time of writing, received any response from upstream about this, unfortunately. I uploaded the fixed packages as DLA 1150-1 on October 31st.

Git-annex The next big chunk on my list was completing the work on git-annex (CVE-2017-12976) that I started in August. It turns out doing the backport was simpler than I expected, even with my rusty experience with Haskell. Type-checking really helps in doing the right thing, especially considering how Joey Hess implemented the fix: by introducing a new type. So I backported the patch from upstream and notified the security team that the jessie and stretch updates would be similarly easy. I shipped the backport to LTS as DLA-1144-1. I also shared the updated packages for jessie (which required a similar backport) and stretch (which didn't) and those Sebastien Delafond published those as DSA 4010-1.

Graphicsmagick Up next was yet another security vulnerability in the Graphicsmagick stack. This involved the usual deep dive into intricate and sometimes just unreasonable C code to try and fit a round tree in a square sinkhole. I'm always unsure about those patches, but the test suite passes, smoke tests show the vulnerability as fixed, and that's pretty much as good as it gets. The announcement (DLA 1154-1) turned out to be a little special because I had previously noticed that the penultimate announcement (DLA 1130-1) was never sent out. So I made a merged announcement to cover both instead of re-sending the original 3 weeks late, which may have been confusing for our users.

Triage & misc We always do a bit of triage even when not on frontdesk duty, so I: I also did smaller bits of work on: The latter reminded me of the concerns I have about the long-term maintainability of the golang ecosystem: because everything is statically linked, an update to a core library (say the SMTP library as in CVE-2017-15042, thankfully not affecting LTS) requires a full rebuild of all packages including the library in all distributions. So what would be a simple update in a shared library system could mean an explosion of work on statically linked infrastructures. This is a lot of work which can definitely be error-prone: as I've seen in other updates, some packages (for example the Ruby interpreter) just bit-rot on their own and eventually fail to build from source. We would also have to investigate all packages to see which one include the library, something which we are not well equipped for at this point. Wheezy was the first release shipping golang packages but at least it's shipping only one... Stretch has shipped with two golang versions (1.7 and 1.8) which will make maintenance ever harder in the long term.
We build our computers the way we build our cities--over time, without a plan, on top of ruins. - Ellen Ullman

Other free software work This month again, I was busy doing some serious yak shaving operations all over the internet, on top of publishing two of my largest LWN articles to date (2017-10-16-strategies-offline-pgp-key-storage and 2017-10-26-comparison-cryptographic-keycards).

feed2exec beta Since I announced this new project last month I have released it as a beta and it entered Debian. I have also wrote useful plugins like the wayback plugin that saves pages on the Wayback machine for eternal archival. The archive plugin can also similarly save pages to the local filesystem. I also added bash completion, expanded unit tests and documentation, fixed default file paths and a bunch of bugs, and refactored the code. Finally, I also started using two external Python libraries instead of rolling my own code: the pyxdg and requests-file libraries, the latter which I packaged in Debian (and fixed a bug in their test suite). The program is working pretty well for me. The only thing I feel is really missing now is a retry/fail mechanism. Right now, it's a little brittle: any network hiccup will yield an error email, which are readable to me but could be confusing to a new user. Strangely enough, I am particularly having trouble with (local!) DNS resolution that I need to look into, but that is probably unrelated with the software itself. Thankfully, the user can disable those with --loglevel=ERROR to silence WARNINGs. Furthermore, some plugins still have some rough edges. For example, The Transmission integration would probably work better as a distinct plugin instead of a simple exec call, because when it adds new torrents, the output is totally cryptic. That plugin could also leverage more feed parameters to save different files in different locations depending on the feed titles, something would be hard to do safely with the exec plugin now. I am keeping a steady flow of releases. I wish there was a way to see how effective I am at reaching out with this project, but unfortunately GitLab doesn't provide usage statistics... And I have received only a few comments on IRC about the project, so maybe I need to reach out more like it says in the fine manual. Always feels strange to have to promote your project like it's some new bubbly soap... Next steps for the project is a final review of the API and release production-ready 1.0.0. I am also thinking of making a small screencast to show the basic capabilities of the software, maybe with asciinema's upcoming audio support?

Pandoc filters As I mentioned earlier, I dove again in Haskell programming when working on the git-annex security update. But I also have a small Haskell program of my own - a Pandoc filter that I use to convert the HTML articles I publish on LWN.net into a Ikiwiki-compatible markdown version. It turns out the script was still missing a bunch of stuff: image sizes, proper table formatting, etc. I also worked hard on automating more bits of the publishing workflow by extracting the time from the article which allowed me to simply extract the full article into an almost final copy just by specifying the article ID. The only thing left is to add tags, and the article is complete. In the process, I learned about new weird Haskell constructs. Take this code, for example:
-- remove needless blockquote wrapper around some tables
--
-- haskell newbie tips:
--
-- @ is the "at-pattern", allows us to define both a name for the
-- construct and inspect the contents as once
--
--   is the "empty record pattern": it basically means "match the
-- arguments but ignore the args"
cleanBlock (BlockQuote t@[Table  ]) = t
Here the idea is to remove <blockquote> elements needlessly wrapping a <table>. I can't specify the Table type on its own, because then I couldn't address the table as a whole, only its parts. I could reconstruct the whole table bits by bits, but it wasn't as clean. The other pattern was how to, at last, address multiple string elements, which was difficult because Pandoc treats spaces specially:
cleanBlock (Plain (Strong (Str "Notifications":Space:Str "for":Space:Str "all":Space:Str "responses":_):_)) = []
The last bit that drove me crazy was the date parsing:
-- the "GAByline" div has a date, use it to generate the ikiwiki dates
--
-- this is distinct from cleanBlock because we do not want to have to
-- deal with time there: it is only here we need it, and we need to
-- pass it in here because we do not want to mess with IO (time is I/O
-- in haskell) all across the function hierarchy
cleanDates :: ZonedTime -> Block -> [Block]
-- this mouthful is just the way the data comes in from
-- LWN/Pandoc. there could be a cleaner way to represent this,
-- possibly with a record, but this is complicated and obscure enough.
cleanDates time (Div (_, [cls], _)
                 [Para [Str month, Space, Str day, Space, Str year], Para _])
    cls == "GAByline" = ikiwikiRawInline (ikiwikiMetaField "date"
                                           (iso8601Format (parseTimeOrError True defaultTimeLocale "%Y-%B-%e,"
                                                           (year ++ "-" ++ month ++ "-" ++ day) :: ZonedTime)))
                        ++ ikiwikiRawInline (ikiwikiMetaField "updated"
                                             (iso8601Format time))
                        ++ [Para []]
-- other elements just pass through
cleanDates time x = [x]
Now that seems just dirty, but it was even worse before. One thing I find difficult in adapting to coding in Haskell is that you need to take the habit of writing smaller functions. The language is really not well adapted to long discourse: it's more about getting small things connected together. Other languages (e.g. Python) discourage this because there's some overhead in calling functions (10 nanoseconds in my tests, but still), whereas functions are a fundamental and important construction in Haskell that are much more heavily optimized. So I constantly need to remind myself to split things up early, otherwise I can't do anything in Haskell. Other languages are more lenient, which does mean my code can be more dirty, but I feel get things done faster then. The oddity of Haskell makes frustrating to work with. It's like doing construction work but you're not allowed to get the floor dirty. When I build stuff, I don't mind things being dirty: I can cleanup afterwards. This is especially critical when you don't actually know how to make things clean in the first place, as Haskell will simply not let you do that at all. And obviously, I fought with Monads, or, more specifically, "I/O" or IO in this case. Turns out that getting the current time is IO in Haskell: indeed, it's not a "pure" function that will always return the same thing. But this means that I would have had to change the signature of all the functions that touched time to include IO. I eventually moved the time initialization up into main so that I had only one IO function and moved that timestamp downwards as simple argument. That way I could keep the rest of the code clean, which seems to be an acceptable pattern. I would of course be happy to get feedback from my Haskell readers (if any) to see how to improve that code. I am always eager to learn.

Git remote MediaWiki Few people know that there is a MediaWiki remote for Git which allow you to mirror a MediaWiki site as a Git repository. As a disaster recovery mechanism, I have been keeping such a historical backup of the Amateur radio wiki for a while now. This originally started as a homegrown Python script to also convert the contents in Markdown. My theory then was to see if we could switch from Mediawiki to Ikiwiki, but it took so long to implement that I never completed the work. When someone had the weird idea of renaming a page to some impossible long name on the wiki, my script broke. I tried to look at fixing it and then remember I also had a mirror running using the Git remote. It turns out it also broke on the same issue and that got me looking in the remote again. I got lost in a zillion issues, including fixing that specific issue, but I especially looked at the possibility of fetching all namespaces because I realized that the remote fetches only a part of the wiki by default. And that drove me to submit namespace support as a patch to the git mailing list. Finally, the discussion came back to how to actually maintain that contrib: in git core or outside? Finally, it looks like I'll be doing some maintenance that project outside of git, as I was granted access to the GitHub organisation...

Galore Yak Shaving Then there's the usual hodgepodge of fixes and random things I did over the month.
There is no [web extension] only XUL! - Inside joke

25 October 2017

Petter Reinholdtsen: Locating IMDB IDs of movies in the Internet Archive using Wikidata

Recently, I needed to automatically check the copyright status of a set of The Internet Movie database (IMDB) entries, to figure out which one of the movies they refer to can be freely distributed on the Internet. This proved to be harder than it sounds. IMDB for sure list movies without any copyright protection, where the copyright protection has expired or where the movie is lisenced using a permissive license like one from Creative Commons. These are mixed with copyright protected movies, and there seem to be no way to separate these classes of movies using the information in IMDB. First I tried to look up entries manually in IMDB, Wikipedia and The Internet Archive, to get a feel how to do this. It is hard to know for sure using these sources, but it should be possible to be reasonable confident a movie is "out of copyright" with a few hours work per movie. As I needed to check almost 20,000 entries, this approach was not sustainable. I simply can not work around the clock for about 6 years to check this data set. I asked the people behind The Internet Archive if they could introduce a new metadata field in their metadata XML for IMDB ID, but was told that they leave it completely to the uploaders to update the metadata. Some of the metadata entries had IMDB links in the description, but I found no way to download all metadata files in bulk to locate those ones and put that approach aside. In the process I noticed several Wikipedia articles about movies had links to both IMDB and The Internet Archive, and it occured to me that I could use the Wikipedia RDF data set to locate entries with both, to at least get a lower bound on the number of movies on The Internet Archive with a IMDB ID. This is useful based on the assumption that movies distributed by The Internet Archive can be legally distributed on the Internet. With some help from the RDF community (thank you DanC), I was able to come up with this query to pass to the SPARQL interface on Wikidata:
SELECT ?work ?imdb ?ia ?when ?label
WHERE
 
  ?work wdt:P31/wdt:P279* wd:Q11424.
  ?work wdt:P345 ?imdb.
  ?work wdt:P724 ?ia.
  OPTIONAL  
        ?work wdt:P577 ?when.
        ?work rdfs:label ?label.
        FILTER(LANG(?label) = "en").
   
 
If I understand the query right, for every film entry anywhere in Wikpedia, it will return the IMDB ID and The Internet Archive ID, and when the movie was released and its English title, if either or both of the latter two are available. At the moment the result set contain 2338 entries. Of course, it depend on volunteers including both correct IMDB and The Internet Archive IDs in the wikipedia articles for the movie. It should be noted that the result will include duplicates if the movie have entries in several languages. There are some bogus entries, either because The Internet Archive ID contain a typo or because the movie is not available from The Internet Archive. I did not verify the IMDB IDs, as I am unsure how to do that automatically. I wrote a small python script to extract the data set from Wikidata and check if the XML metadata for the movie is available from The Internet Archive, and after around 1.5 hour it produced a list of 2097 free movies and their IMDB ID. In total, 171 entries in Wikidata lack the refered Internet Archive entry. I assume the 70 "disappearing" entries (ie 2338-2097-171) are duplicate entries. This is not too bad, given that The Internet Archive report to contain 5331 feature films at the moment, but it also mean more than 3000 movies are missing on Wikipedia or are missing the pair of references on Wikipedia. I was curious about the distribution by release year, and made a little graph to show how the amount of free movies is spread over the years: I expect the relative distribution of the remaining 3000 movies to be similar. If you want to help, and want to ensure Wikipedia can be used to cross reference The Internet Archive and The Internet Movie Database, please make sure entries like this are listed under the "External links" heading on the Wikipedia article for the movie:
*  Internet Archive film id=FightingLady 
*  IMDb title id=0036823 title=The Fighting Lady 
Please verify the links on the final page, to make sure you did not introduce a typo. Here is the complete list, if you want to correct the 171 identified Wikipedia entries with broken links to The Internet Archive: Q1140317, Q458656, Q458656, Q470560, Q743340, Q822580, Q480696, Q128761, Q1307059, Q1335091, Q1537166, Q1438334, Q1479751, Q1497200, Q1498122, Q865973, Q834269, Q841781, Q841781, Q1548193, Q499031, Q1564769, Q1585239, Q1585569, Q1624236, Q4796595, Q4853469, Q4873046, Q915016, Q4660396, Q4677708, Q4738449, Q4756096, Q4766785, Q880357, Q882066, Q882066, Q204191, Q204191, Q1194170, Q940014, Q946863, Q172837, Q573077, Q1219005, Q1219599, Q1643798, Q1656352, Q1659549, Q1660007, Q1698154, Q1737980, Q1877284, Q1199354, Q1199354, Q1199451, Q1211871, Q1212179, Q1238382, Q4906454, Q320219, Q1148649, Q645094, Q5050350, Q5166548, Q2677926, Q2698139, Q2707305, Q2740725, Q2024780, Q2117418, Q2138984, Q1127992, Q1058087, Q1070484, Q1080080, Q1090813, Q1251918, Q1254110, Q1257070, Q1257079, Q1197410, Q1198423, Q706951, Q723239, Q2079261, Q1171364, Q617858, Q5166611, Q5166611, Q324513, Q374172, Q7533269, Q970386, Q976849, Q7458614, Q5347416, Q5460005, Q5463392, Q3038555, Q5288458, Q2346516, Q5183645, Q5185497, Q5216127, Q5223127, Q5261159, Q1300759, Q5521241, Q7733434, Q7736264, Q7737032, Q7882671, Q7719427, Q7719444, Q7722575, Q2629763, Q2640346, Q2649671, Q7703851, Q7747041, Q6544949, Q6672759, Q2445896, Q12124891, Q3127044, Q2511262, Q2517672, Q2543165, Q426628, Q426628, Q12126890, Q13359969, Q13359969, Q2294295, Q2294295, Q2559509, Q2559912, Q7760469, Q6703974, Q4744, Q7766962, Q7768516, Q7769205, Q7769988, Q2946945, Q3212086, Q3212086, Q18218448, Q18218448, Q18218448, Q6909175, Q7405709, Q7416149, Q7239952, Q7317332, Q7783674, Q7783704, Q7857590, Q3372526, Q3372642, Q3372816, Q3372909, Q7959649, Q7977485, Q7992684, Q3817966, Q3821852, Q3420907, Q3429733, Q774474

20 October 2017

Rapha&#235;l Hertzog: Freexian s report about Debian Long Term Support, September 2017

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS. Individual reports In August, about 170 work hours have been dispatched among 13 paid contributors. Their reports are available: Evolution of the situation The number of sponsored hours is the same as last month. But we have a new sponsor in the pipe. The security tracker currently lists 52 packages with a known CVE and the dla-needed.txt file 49. The number of packages with open issues decreased slightly compared to last month but we re not yet back to the usual situation. Thanks to our sponsors New sponsors are in bold.

No comment Liked this article? Click here. My blog is Flattr-enabled.

17 October 2017

Norbert Preining: Japanese TeX User Meeting 2017

Last saturday the Japanese TeX User Meeting took place in Fujisawa, Kanagawa. For those who have been at the TUG 2013 in Tokyo you will remember that the Japanese TeX community is quite big and vibrant. On Saturday about 50 users and developers gathered for a set of talks on a variety of topics. The first talk was by Keiichiro Shikano ( ) on using Markup text to generate (La)TeX and HTML. He presented a variety of markup formats, including his own tool xml2tex. The second talk was my Masamichi Hosoda ( ) on reducing the size of PDF files using PDFmark extraction. As a contributor to many projects including Texinfo and LilyPond, Masamichi Hosoda tells us horror stories about multiple font embedding in the manual of LilyPond, the permanent need for adaption to newer Ghostscript versions, and the very recent development in Ghostscript prohibiting the merge of font definitions in PDF files. Next up was Yusuke Terada ( ) on grading exams using TeX. Working through hundreds and hundreds of exams and do the grading is something many of us are used to and I think nobody really enjoys it. Yusuke Terada has combined various tools, including scans, pdf merging using pdfpages, to generate gradable PDF which were then checked on an iPad. On the way he did hit some limits in dvipdfmx on the number of images, but this was obviously only a small bump on the road. Now if that could be automatized as a nice application, it would be a big hit I guess! The forth talk was by Satoshi Yamashita ( ) on the preparation of slides using KETpic. KETpic is a long running project by Setsuo Takato ( ) for the generation of graphics, in particular using Cinderella. KETpic and KETcindy integrates with lots of algebraic and statistical programs (R, Maxima, SciLab, ) and has a long history of development. Currently there are activities to incorporate it into TeX Live. The fifth talk was by Takuto Asakura ( ) on programming TeX using expl3, the main building block of the LaTeX3 project and already adopted by many TeX developers. Takuto Asakura came to fame on this years TUG/BachoTeX 2017 when he won the W. J. Martin Prize for his presentation Implementing bioinformatics algorithms in TeX. I think we can expect great new developments from Takuto! The last talk was by myself on fmtutil and updmap, two of the main management programs in any TeX installation, presenting the changes introduced over the last year, including the most recent release of TeX Live. Details have been posted on my blog, and a lengthy article in TUGboat 38:2, 2017 is available on this topic, too. After the conference about half of the participants joined a social dinner in a nearby Izakaya, followed by a after-dinner beer tasting at a local craft beer place. Thanks to Tatsuyoshi Hamada for the organization. As usual, the Japanese TeX User Meetings are a great opportunity to discuss new features and make new friends. I am always grateful to be part of this very nice community! I am looking forward to the next year s meeting.

16 October 2017

Gustavo Noronha Silva: Who knew we still had low-hanging fruits?

Earlier this month I had the pleasure of attending the Web Engines Hackfest, hosted by Igalia at their offices in A Coru a, and also sponsored by my employer, Collabora, Google and Mozilla. It has grown a lot and we had many new people this year. Fun fact: I am one of the 3 or 4 people who have attended all of the editions of the hackfest since its inception in 2009, when it was called WebKitGTK+ hackfest \o/ 20171002_204405 It was a great get together where I met many friends and made some new ones. Had plenty of discussions, mainly with Antonio Gomes and Google s Robert Kroeger, about the way forward for Chromium on Wayland. We had the opportunity of explaining how we at Collabora cooperated with igalians to implemented and optimise a Wayland nested compositor for WebKit2 to share buffers between processes in an efficient way even on broken drivers. Most of the discussions and some of the work that led to this was done in previous hackfests, by the way! 20171002_193518 The idea seems to have been mostly welcomed, the only concern being that Wayland s interfaces would need to be tested for security (fuzzed). So we may end up going that same route with Chromium for allowing process separation between the UI and GPU (being renamed Viz, currently) processes. On another note, and going back to the title of the post, at Collabora we have recently adopted Mattermost to replace our internal IRC server. Many Collaborans have decided to use Mattermost through an Epiphany Web Application or through a simple Python application that just shows a GTK+ window wrapping a WebKitGTK+ WebView. 20171002_101952 Some people noticed that when the connection was lost Mattermost would take a very long time to notice and reconnect its web sockets were taking a long, long time to timeout, according to our colleague Andrew Shadura. I did some quick searching on the codebase and noticed WebCore has a NetworkStateNotifier interface that it uses to get notified when connection changes. That was not implemented for WebKitGTK+, so it was likely what caused stuff to linger when a connection hiccup happened. Given we have GNetworkMonitor, implementation of the missing interfaces required only 3 lines of actual code (plus the necessary boilerplate)! screenshot-from-2017-10-16-11-13-39 I was surprised to still find such as low hanging fruit in WebKitGTK+, so I decided to look for more. Turns out WebCore also has a notifier for low power situations, which was implemented only by the iOS port, and causes the engine to throttle some timers and avoid some expensive checks it would do in normal situations. This required a few more lines to implement using upower-glib, but not that many either! That was the fun I had during the hackfest in terms of coding. Mostly I had fun just lurking in break out sessions discussing the past, present and future of tech such as WebRTC, Servo, Rust, WebKit, Chromium, WebVR, and more. I also beat a few challengers in Street Fighter 2, as usual. I d like to say thanks to Collabora, Igalia, Google, and Mozilla for sponsoring and attending the hackfest. Thanks to Igalia for hosting and to Collabora for sponsoring my attendance along with two other Collaborans. It was a great hackfest and I m looking forward to the next one! See you in 2018 =)

12 October 2017

Joachim Breitner: Isabelle functions: Always total, sometimes undefined

Often, when I mention how things work in the interactive theorem prover [Isabelle/HOL] (in the following just Isabelle 1) to people with a strong background in functional programming (whether that means Haskell or Coq or something else), I cause confusion, especially around the issue of what is a function, are function total and what is the business with undefined. In this blog post, I want to explain some these issues, aimed at functional programmers or type theoreticians. Note that this is not meant to be a tutorial; I will not explain how to do these things, and will focus on what they mean.

HOL is a logic of total functions If I have a Isabelle function f :: a b between two types a and b (the function arrow in Isabelle is , not ), then by definition of what it means to be a function in HOL whenever I have a value x :: a, then the expression f x (i.e. f applied to x) is a value of type b. Therefore, and without exception, every Isabelle function is total. In particular, it cannot be that f x does not exist for some x :: a. This is a first difference from Haskell, which does have partial functions like
spin :: Maybe Integer -> Bool
spin (Just n) = spin (Just (n+1))
Here, neither the expression spin Nothing nor the expression spin (Just 42) produce a value of type Bool: The former raises an exception ( incomplete pattern match ), the latter does not terminate. Confusingly, though, both expressions have type Bool. Because every function is total, this confusion cannot arise in Isabelle: If an expression e has type t, then it is a value of type t. This trait is shared with other total systems, including Coq. Did you notice the emphasis I put on the word is here, and how I deliberately did not write evaluates to or returns ? This is because of another big source for confusion:

Isabelle functions do not compute We (i.e., functional programmers) stole the word function from mathematics and repurposed it2. But the word function , in the context of Isabelle, refers to the mathematical concept of a function, and it helps to keep that in mind. What is the difference?
  • A function a b in functional programming is an algorithm that, given a value of type a, calculates (returns, evaluates to) a value of type b.
  • A function a b in math (or Isabelle) associates with each value of type a a value of type b.
For example, the following is a perfectly valid function definition in math (and HOL), but could not be a function in the programming sense:
definition foo :: "(nat   real)   real" where
  "foo seq = (if convergent seq then lim seq else 0)"
This assigns a real number to every sequence, but it does not compute it in any useful sense. From this it follows that

Isabelle functions are specified, not defined Consider this function definition:
fun plus :: "nat   nat   nat"  where
   "plus 0       m = m"
   "plus (Suc n) m = Suc (plus n m)"
To a functional programmer, this reads
plus is a function that analyses its first argument. If that is 0, then it returns the second argument. Otherwise, it calls itself with the predecessor of the first argument and increases the result by one.
which is clearly a description of a computation. But to Isabelle, the above reads
plus is a binary function on natural numbers, and it satisfies the following two equations:
And in fact, it is not so much Isabelle that reads it this way, but rather the fun command, which is external to the Isabelle logic. The fun command analyses the given equations, constructs a non-recursive definition of plus under the hood, passes that to Isabelle and then proves that the given equations hold for plus. One interesting consequence of this is that different specifications can lead to the same functions. In fact, if we would define plus' by recursing on the second argument, we d obtain the the same function (i.e. plus = plus' is a theorem, and there would be no way of telling the two apart).

Termination is a property of specifications, not functions Because a function does not evaluate, it does not make sense to ask if it terminates. The question of termination arises before the function is defined: The fun command can only construct plus in a way that the equations hold if it passes a termination check very much like Fixpoint in Coq. But while the termination check of Fixpoint in Coq is a deep part of the basic logic, in Isabelle it is simply something that this particular command requires for its internal machinery to go through. At no point does a termination proof of the function exist as a theorem inside the logic. And other commands may have other means of defining a function that do not even require such a termination argument! For example, a function specification that is tail-recursive can be turned in to a function, even without a termination proof: The following definition describes a higher-order function that iterates its first argument f on the second argument x until it finds a fixpoint. It is completely polymorphic (the single quote in 'a indicates that this is a type variable):
partial_function (tailrec)
  fixpoint :: "('a   'a)   'a   'a"
where
  "fixpoint f x = (if f x = x then x else fixpoint f (f x))"
We can work with this definition just fine. For example, if we instantiate f with ( x. x-1), we can prove that it will always return 0:
lemma "fixpoint (  n . n - 1) (n::nat) = 0"
  by (induction n) (auto simp add: fixpoint.simps)
Similarly, if we have a function that works within the option monad (i.e. Maybe in Haskell), its specification can always be turned into a function without an explicit termination proof here one that calculates the Collatz sequence:
partial_function (option) collatz :: "nat   nat list option"
 where "collatz n =
        (if n = 1 then Some [n]
         else if even n
           then do   ns <- collatz (n div 2);    Some (n # ns)  
           else do   ns <- collatz (3 * n + 1);  Some (n # ns) )"
Note that lists in Isabelle are finite (like in Coq, unlike in Haskell), so this function returns a list only if the collatz sequence eventually reaches 1. I expect these definitions to make a Coq user very uneasy. How can fixpoint be a total function? What is fixpoint ( n. n+1)? What if we run collatz n for a n where the Collatz sequence does not reach 1?3 We will come back to that question after a little detour

HOL is a logic of non-empty types Another big difference between Isabelle and Coq is that in Isabelle, every type is inhabited. Just like the totality of functions, this is a very fundamental fact about what HOL defines to be a type. Isabelle gets away with that design because in Isabelle, we do not use types for propositions (like we do in Coq), so we do not need empty types to denote false propositions. This design has an important consequence: It allows the existence of a polymorphic expression that inhabits any type, namely
undefined :: 'a
The naming of this term alone has caused a great deal of confusion for Isabelle beginners, or in communication with users of different systems, so I implore you to not read too much into the name. In fact, you will have a better time if you think of it as arbitrary or, even better, unknown. Since undefined can be instantiated at any type, we can instantiate it for example at bool, and we can observe an important fact: undefined is not an extra value besides the usual ones . It is simply some value of that type, which is demonstrated in the following lemma:
lemma "undefined = True   undefined = False" by auto
In fact, if the type has only one value (such as the unit type), then we know the value of undefined for sure:
lemma "undefined = ()" by auto
It is very handy to be able to produce an expression of any type, as we will see as follows

Partial functions are just underspecified functions For example, it allows us to translate incomplete function specifications. Consider this definition, Isabelle s equivalent of Haskell s partial fromJust function:
fun fromSome :: "'a option   'a" where
  "fromSome (Some x) = x"
This definition is accepted by fun (albeit with a warning), and the generated function fromSome behaves exactly as specified: when applied to Some x, it is x. The term fromSome None is also a value of type 'a, we just do not know which one it is, as the specification does not address that. So fromSome None behaves just like undefined above, i.e. we can prove
lemma "fromSome None = False   fromSome None = True" by auto
Here is a small exercise for you: Can you come up with an explanation for the following lemma:
fun constOrId :: "bool   bool" where
  "constOrId True = True"
lemma "constOrId = ( _.True)   constOrId = ( x. x)"
  by (metis (full_types) constOrId.simps)
Overall, this behavior makes sense if we remember that function definitions in Isabelle are not really definitions, but rather specifications. And a partial function definition is simply a underspecification. The resulting function is simply any function hat fulfills the specification, and the two lemmas above underline that observation.

Nonterminating functions are also just underspecified Let us return to the puzzle posed by fixpoint above. Clearly, the function seen as a functional program is not total: When passed the argument ( n. n + 1) or ( b. b) it will loop forever trying to find a fixed point. But Isabelle functions are not functional programs, and the definitions are just specifications. What does the specification say about the case when f has no fixed-point? It states that the equation fixpoint f x = fixpoint f (f x) holds. And this equation has a solution, for example fixpoint f _ = undefined. Or more concretely: The specification of the fixpoint function states that fixpoint ( b. b) True = fixpoint ( b. b) False has to hold, but it does not specify which particular value (True or False) it should denote any is fine.

Not all function specifications are ok At this point you might wonder: Can I just specify any equations for a function f and get a function out of that? But rest assured: That is not the case. For example, no Isabelle command allows you define a function bogus :: () nat with the equation bogus () = Suc (bogus ()), because this equation does not have a solution. We can actually prove that such a function cannot exist:
lemma no_bogus: "  bogus. bogus () = Suc (bogus ())" by simp
(Of course, not_bogus () = not_bogus () is just fine )

You cannot reason about partiality in Isabelle We have seen that there are many ways to define functions that one might consider partial . Given a function, can we prove that it is not partial in that sense? Unfortunately, but unavoidably, no: Since undefined is not a separate, recognizable value, but rather simply an unknown one, there is no way of stating that A function result is not specified . Here is an example that demonstrates this: Two partial functions (one with not all cases specified, the other one with a self-referential specification) are indistinguishable from the total variant:
fun partial1 :: "bool   unit" where
  "partial1 True = ()"
partial_function (tailrec) partial2 :: "bool   unit" where
  "partial2 b = partial2 b"
fun total :: "bool   unit" where
  "total True = ()"
  "total False = ()"
lemma "partial1 = total   partial2 = total" by auto
If you really do want to reason about partiality of functional programs in Isabelle, you should consider implementing them not as plain HOL functions, but rather use HOLCF, where you can give equational specifications of functional programs and obtain continuous functions between domains. In that setting, () and partial2 = total. We have done that to verify some of HLint s equations.

You can still compute with Isabelle functions I hope by this point, I have not scared away anyone who wants to use Isabelle for functional programming, and in fact, you can use it for that. If the equations that you pass to fun are a reasonable definition for a function (in the programming sense), then these equations, used as rewriting rules, will allow you to compute that function quite like you would in Coq or Haskell. Moreover, Isabelle supports code extraction: You can take the equations of your Isabelle functions and have them expored into Ocaml, Haskell, Scala or Standard ML. See Concon for a conference management system with confidentially verified in Isabelle. While these usually are the equations you defined the function with, they don't have to: You can declare other proved equations to be used for code extraction, e.g. to refine your elegant definitions to performant ones. Like with code extraction from Coq to, say, Haskell, the adequacy of the translations rests on a moral reasoning foundation. Unlike extraction from Coq, where you have an (unformalized) guarantee that the resulting Haskell code is terminating, you do not get that guarantee from Isabelle. Conversely, this allows you do reason about and extract non-terminating programs, like fixpoint, which is not possible in Coq. There is currently ongoing work about verified code generation, where the code equations are reflected into a deep embedding of HOL in Isabelle that would allow explicit termination proofs.

Conclusion We have seen how in Isabelle, every function is total. Function declarations have equations, but these do not define the function in an computational sense, but rather specify them. Because in HOL, there are no empty types, many specifications that appear partial (incomplete patterns, non-terminating recursion) have solutions in the space of total functions. Partiality in the specification is no longer visible in the final product.

PS: Axiom undefined in Coq This section is speculative, and an invitation for discussion. Coq already distinguishes between types used in programs (Set) and types used in proofs Prop. Could Coq ensure that every t : Set is non-empty? I imagine this would require additional checks in the Inductive command, similar to the checks that the Isabelle command datatype has to perform4, and it would disallow Empty_set. If so, then it would be sound to add the following axiom
Axiom undefined : forall (a : Set), a.
wouldn't it? This axiom does not have any computational meaning, but that seems to be ok for optional Coq axioms, like classical reasoning or function extensionality. With this in place, how much of what I describe above about function definitions in Isabelle could now be done soundly in Coq. Certainly pattern matches would not have to be complete and could sport an implicit case _ undefined. Would it help with non-obviously terminating functions? Would it allow a Coq command Tailrecursive that accepts any tailrecursive function without a termination check?

  1. Isabelle is a metalogical framework, and other logics, e.g. Isabelle/ZF, behave differently. For the purpose of this blog post, I always mean Isabelle/HOL.
  2. Isabelle is a metalogical framework, and other logics, e.g. Isabelle/ZF, behave differently. For the purpose of this blog post, I always mean Isabelle/HOL.
  3. Let me know if you find such an n. Besides n = 0.
  4. Like fun, the constructions by datatype are not part of the logic, but create a type definition from more primitive notions that is isomorphic to the specified data type.

Next.