Search Results: "ag"

7 September 2020

Arturo Borrero Gonz lez: Debconf 2020 online, summary

Debconf2020 logo Debconf2020 took place when I was on personal vacations time. But anyway I m lucky enough that my company, the Wikimedia Foundation, paid the conference registration fee for me and allowed me to take the time (after my vacations) to watch recordings from the conference. This is my first time attending (or watching) a full-online conference, and I was curious to see first hand how it would develop. I was greatly surprised to see it worked pretty nicely, so kudos to the organization, video team, volunteers, etc! What follows is my summary of the conference, from the different sessions and talks I watched (again, none of them live but recordings). The first thing I saw was the Welcome to Debconf 2020 opening session. It is obvious the video was made with lots of love, I found it entertaining and useful. I love it :-) Then I watched the BoF Can Free Software improve social equality. It was introduced and moderated by Hong Phuc Dang. Several participants, about 10 people, shared their visions on the interaction between open source projects and communities. I m pretty much aware of the interesting social advancement that FLOSS can enable in communities, but sometimes is not so easy, it may also present challenges and barriers. The BoF was joined by many people from the Asia Pacific region, and for me, it has been very interesting to take a step back from the usual western vision of this topic. Anyway, about the session itself, I have the feeling the participants may have spent too much time on presentations, sharing their local stories (which are interesting, don t get me wrong), perhaps leaving little room for actual proposal discussions or the like. Next I watched the Bits from the DPL talk. In the session, Jonathan Carter goes over several topics affecting the project, both internally and externally. It was interesting to know more about the status of the project from a high level perspective, as an organization, including subjects such as money, common project problems, future issues we are anticipating, the social aspect of the project, etc. The Lightning Talks session grabbed my attention. It is usually very funny to watch and not as dense as other talks. I m glad I watched this as it includes some interesting talks, ranging from HAM radios (I love them!), to personal projects to help in certain tasks, and even some general reflections about life. Just when I m writing this very sentence, the video for the Come and meet your Debian Publicity team! talk has been uploaded. This team does an incredible work in keeping project information flowing, and social networks up-to-date and alive. Mind that the work of this team is mostly non-engineering, but still, is a vital part of the project. The folks in session explain what the team does, and they also discuss how new people can contribute, the different challenges related to language barriers, etc. I have to admit I also started watching a couple other sessions that turned out to don t be interesting to me (and therefore I didn t finish the video). Also, I tried to watch a couple more sessions that didn t publish their video recording just yet, for example the When We Virtualize the Whole Internet talk by Sam Hartman. Will check again in a couple of days. It is a real pleasure the video recordings from the conference are made available online. One can join the conference anytime (like I m doing!) and watch the sessions at any pace at any time. The video archive is big, I won t be able to go over all of it. I won t lie, I still have some pending videos to watch from last year Debconf2019 :-)

6 September 2020

Jonathan Carter: DebConf 20 Online

This week, last week, Last month, I attended DebConf 20 Online. It was the first DebConf to be held entirely online, but it s the 7th DebConf I ve attended from home. My first one was DebConf7. Initially I mostly started watching the videos because I wanted to learn more about packaging. I had just figured out how to create binary packages by hand, and have read through the new maintainers guide, but a lot of it was still a mystery. By the end of DebConf7 my grasp of source packages was still a bit thin, but other than that, I ended up learning a lot more about Debian during DebConf7 than I had hoped for, and over the years, the quality of online participation for each DebConf has varied a lot. I think having a completely online DebConf, where everyone was remote, helped raise awareness about how important it is to make the remote experience work well, and I hope that it will make people who run sessions at physical events in the future consider those who are following remotely a bit more. During some BoF sessions, it was clear that some teams haven t talked to each other face to face in a while, and I heard at least 3 teams who said This was nice, we should do more regular video calls! . Our usual communication methods of e-mail lists and IRC serve us quite well, for the most part, but sometimes having an actual conversation with the whole team present at the same time can do wonders for dealing with many kind of issues that is just always hard to deal with in text based mediums. There were three main languages used in this DebConf. We ve had more than one language at a DebConf before, but as far as I know it s the first time that we had multiple talks over 3 languages (English, Malayalam and Spanish). It was also impressive how the DebConf team managed to send out DebConf t-shirts all around the world and in time before the conference! To my knowledge only 2 people didn t get theirs in time due to customs. I already posted about the new loop that we worked on for this DebConf. It was an unintended effect that we ended up having lots of shout-outs which ended up giving this online DebConf a much more warmer, personal feel to it than if we didn t have it. I m definitely planning to keep on improving on that for the future, for online and in-person events. There were also some other new stuff from the video team during this DebConf, we ll try to co-ordinate a blog post about that once the dust settled. Thanks to everyone for making this DebConf special, even though it was virtual!

Thorsten Alteholz: My Debian Activities in August 2020

FTP master This month I accepted 159 packages and rejected 16. The overall number of packages that got accepted was 172. Debian LTS This was my seventy-fourth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. This month my all in all workload has been 21.75h. During that time I did LTS uploads of: I also started to work on curl but did not upload a fixed version yet. As usual, testing the package takes up some time. Last but not least I did some days of frontdesk duties. Debian ELTS This month was the twenty sixth ELTS month. During my allocated time I uploaded: Like in LTS, I also started to work on curl and encountered the same problems as in LTS above. Last but not least I did some days of frontdesk duties. Other stuff This month I found again some time for other Debian work and uploaded packages to fix bugs, mainly around gcc10: I also uploaded new upstream versions of: All package called *osmo* are developed by the Osmocom project, that is about Open Source MObile COMmunication. They are really doing a great job and I apologize that my uploads of new versions are mostly far behind their development. Some of the uploads are related to new packages:

Dirk Eddelbuettel: inline 0.3.16: Now with system2()

A new minor release of the inline package just arrived on CRAN. inline facilitates writing code in-line in simple string expressions or short files. The package is mature and stable, and can be considered to be in maintenance mode: Rcpp used it extensively in the vrey early days before Rcpp Attributes provided an even better alternative. Seveal other package still rely on inline. One of these package is rstan, and Ben Goodrich updated our use of system() to system2() allowing for better error diagnostics. We also did a bit of standard maintenance to Travis CI and the README.md file. See below for a detailed list of changes extracted from the NEWS file.

Changes in inline version 0.3.16 (2020-09-06)
  • Maintenance updates to README.md standardizing badges (Dirk).
  • Maintenance update to Travis CI setup (Dirk).
  • Switch to using system2() for better error diagnostics (Ben Goodrich in #12).

Courtesy of CRANberries, there is a comparison to the previous release. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Molly de Blanc: NYU VPN

I needed to setup a VPN in order to access my readings for class. The instructions for Linux are located: https://nyu.service-now.com/sp?id=kb_article_view&sysparm_article=KB0014932 After you download the VPN client of your choice (they recommend Cisco AnyConnect), connect to: vpn.nyu.edu. It will ask for two passwords: your NYU username and password and a multi-factor authentication (MFA) code from Duo. Use the Duo. See below for stuff on Duo. Hit connect and viola, you can connect to the VPN.

Duo Authentication Setup Go to: https://start.nyu.edu and follow the instructions for MFA. They ll tell you that a smart phone is the most secure method of setting up. I am skeptical. Install the Duo Authentication App on your phone, enter your phone number into the NYU web page (off of ) and it will send a thing to your phone to connect it. Commentary Okay, I have to complain at least a little bit about this. I had to guess what the VPN address was because the instructions are for NYU Shanghai. I also had to install the VPN client using the terminal. These sorts of things make it harder for people to use Linux. Boo.

Ben Armstrong: Dronefly relicensed under copyleft licenses

To ensure Dronefly always remains free, the Dronefly project has been relicensed under two copyleft licenses. Read the license change and learn more about copyleft at these links. I was prompted to make this change after a recent incident in the Red DiscordBot development community that made me reconsider my prior position that the liberal MIT license was best for our project. While on the face of it, making your license as liberal as possible might seem like the most generous and hassle-free way to license any project, I was shocked into the realization that its liberality was also its fatal flaw: all is well and good so long as everyone is being cooperative, but it does not afford any protection to developers or users should things suddenly go sideways in how a project is run. A copyleft license is the best way to avoid such issues. In this incident a sad story of conflict between developers I respect on both sides of the rift, and owe a debt to for what they ve taught me three cogs we had come to depend on suddenly stopped being viable for us to use due to changes to the license & the code. Effectively, those cogs became unsupported and unsupportable. To avoid any such future disaster with the Dronefly project, I started shopping for a new license that would protect developers and users alike from similarly losing support, or losing control of their contributions. I owe thanks to Dale Floer, a team member who early on advised me the AGPL might be a better fit, and later was helpful in selecting the doc license and encouraging me to follow through. We ran the new licenses by each contributor and arrived at this consensus: the AGPL is best suited for our server-based code, and CC-BY-SA is best suited for our documentation. The relicensing was made official this morning. On Discord platform alternatives You might well question what I, a Debian developer steeped in free software culture and otherwise in agreement with its principles, am doing encouraging a community to grow on the proprietary Discord platform! I have no satisfying answer to that. I explained when I introduced my project here some of the backstory, but that s more of an account of its beginnings than justification for it to continue on this platform. Honestly, all I can offer is a rather dissatisfying it seemed like the right thing to do at the time. Time will tell whether we could successfully move off of it to a freedom-respecting and privacy-respecting alternative chat platform that is both socially and technically viable to migrate to. That platform would ideally: I m intrigued by Matrix and wonder if it provides some or all of the above in its current form. Are you a developer writing bots for this platform? If so, I especially want to hear from you in the comments about your experience. Or in any case, if you ve been there before if you ve faced the same issue with your community and have a success story to share, I would love to hear from you.

Russ Allbery: September haul

So many good books, so little reading time. Jairus Banaji A Brief History of Commercial Capitalism (nonfiction)
Steven Brust The Baron of Magister Valley (sff)
Micaiah Johnson The Space Between Worlds (sff)
Ian McDonald Luna: New Moon (sff)
Elizabeth Moon Trading in Danger (sff)
Tamsyn Muir Harrow the Ninth (sff)
Suzanne Palmer Finder (sff)
Kit Rocha Beyond Shame (sff)
Kit Rocha Beyond Control (sff)
Kit Rocha Beyond Pain (sff)
Arundhati Roy Azadi (nonfiction)
Jeff VanderMeer Authority (sff)
Jeff VanderMeer Acceptance (sff)
K.B. Wagers Behind the Throne (sff)
Jarrett Walker Human Transit (nonfiction) I took advantage of a few sales to get books I know I'm going to want to read eventually for a buck or two.

5 September 2020

Mike Gabriel: My Work on Debian LTS (August 2020)

In August 2020, I have worked on the Debian LTS project for 16 hours (of 8 hours planned, plus another 8 hours that I carried over from July). For ELTS, I have worked for another 8 hours (of 8 hours planned). LTS Work ELTS Work Other security related work for Debian References

Elana Hashman: Three talks at DebConf 2020

This year has been a really unusual one for in-person events like conferences. I had already planned to take this year off from travel for the most part, attending just a handful of domestic conferences. But the pandemic has thrown those plans into chaos; I do not plan to attend large-scale in-person events until July 2021 at the earliest, per my employer's guidance. I've been really sad to have turned down multiple speaking invitations this year. To try to set expectations, I added a note to my Talks page that indicates I will not be writing any new talks for 2020, but am happy to join panels or reprise old talks. And somehow, with all that background, I still ended up giving three talks at DebConf 2020 this year. In part, I think it's because this is the first DebConf I've been able to attend since 2017, and I was so happy to have the opportunity! I took time off work to give myself enough space to focus on the conference. International travel is very difficult for me, so DebConf is generally challenging if not impossible for me to attend. A panel a day keeps the FTP Team away? On Thursday, August 27th, I spoke on the Leadership in Debian panel, where I discussed some of the challenges leadership in the project must face, including an appropriate response to the BLM movement and sustainability for volunteer positions that require unsustainable hours (such as DPL).
On Friday, August 28th, I hosted the Debian Clojure BoF, attended by members of the Clojure and Puppet teams. The Puppet team is working to package the latest versions of Puppet Server/DB, which involve significant Clojure components, and I am doing my best to help.
On Saturday, August 29th, I spoke on the Meet the Technical Committee panel. The Committee presented a number of proposals for improving how we work within the project. I was responsible for presenting our first proposal on allowing folks to engage the committee privately.

4 September 2020

Dirk Eddelbuettel: nanotime 0.3.2: Tweaks

Another (minor) nanotime release, now at version 0.3.2. This release brings an endianness correction which was kindly contributed in a PR, switches to using the API header exported by RcppCCTZ, and tweaks test coverage a little with respect to r-devel. nanotime relies on the RcppCCTZ package for (efficient) high(er) resolution time parsing and formatting up to nanosecond resolution, and the bit64 package for the actual integer64 arithmetic. Initially implemented using the S3 system, it has benefitted greatly from work by co-author Leonardo who not only rejigged nanotime internals in S4 but also added new S4 types for periods, intervals and durations. The NEWS snippet adds full details.

Changes in version 0.3.2 (2020-09-03)
  • Correct for big endian (Elliott Sales de Andrade in #81).
  • Use the RcppCCTZ_API.h header (Dirk in #82).
  • Conditionally reduce test coverage (Dirk in #83).

Thanks to CRANberries there is also a diff to the previous version. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Reproducible Builds (diffoscope): diffoscope 159 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 159. This version includes the following changes: 
[ Chris Lamb ]
* Show "ordering differences only" in strings(1) output.
  (Closes: reproducible-builds/diffoscope#216)
* Don't alias output from "os.path.splitext" to variables that we do not end
  up using.
* Don't raise exceptions when cleaning up after a guestfs cleanup failure.
[ Jean-Romain Garnier ]
* Make "Command" subclass a new generic Operation class.
You find out more by visiting the project homepage.

3 September 2020

Dirk Eddelbuettel: RcppArmadillo 0.9.900.3.0

armadillo image Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language and is widely used by (currently) 769 other packages on CRAN. A few days ago, Conrad released a new minor version 9.900.3 of Armadillo which we packaged and tested as usual. Given the incremental release character, we only tested the release and not candidate release. No regressions were found, and, as usual, logs from reverse-depends runs are in the rcpp-logs repo. All changes in the new release are noted below.

Changes in RcppArmadillo version 0.9.900.3.0 (2020-09-02)
  • Upgraded to Armadillo release 9.900.3 (Nocturnal Misbehaviour)
    • More efficient code for initialising matrices with fill::zeros
    • Fixes for various error messages

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Ben Hutchings: Debian LTS work, August 2020

I was assigned 16 hours of work by Freexian's Debian LTS initiative, but only worked 6.25 hours this month and have carried over the rest to September. I finished my work to add Linux 4.19 to the stretch-security suite, providing an upgrade path for those previously installing it from stretch-backports (DLA-2323-1, DLA-2324-1). I also updated the firmware-nonfree package (DLA-2321-1) so that firmware needed by drivers in Linux 4.19 is also be available in the non-free section of the stretch-security suite. I also reviewed the report of the Debian LTS survey and commented on the presentation of results. This report was presented in the Debian LTS BoF at DebConf.

Molly de Blanc: All Animals Are Equal, Peter Singer

I recently read Disability Visibility, which opens with a piece by Harriet McBryde Johnson about debating Peter Singer. When I got my first reading for my first class and saw it was Peter Singer, I was dismayed because of his (heinous) stances in disability. I assumed All Animals Are Equal was one of Singer s pieces about animal rights. While I agree with many of the principles Singer discusses around animal rights, I feel as though his work on this front is significantly diminished by his work around disability. To put it simply, I can t take Peter Singer seriously. Because of this I had a lot of trouble reading All Animals Are Equal and taking it in good faith. I judged everything from his arguments to his writing harshly. While I don t disagree with his basic point (all animals have rights) I disagree with how he made the point and the argument supporting it. One of the things I was told to ask when reading any philosophy paper is What is the argument? or What are they trying to convince you of? In this case, you could frame the answer as: Animals have some of) the same rights people do. I think it would be more accurate though to frame it as All animals (including humans) have (some of) the same rights or even Humans are as equally worthy of consideration as animals are. I think when we usually talk about animal rights, we do it from a perspective of wanting to elevate animals to human status. From one perspective, I don t like this approach because I feel as though it turns the framing of rights as something you deserve or earn, privileges you get for being good enough. The point about rights is that they are inherent you get them because they are. The valuable thing I got out of All Animals Are Equal is that rights are not universal. When we talk about things like abortion, for example, we talk about the right to have an abortion. Singer asks whether people who cannot get pregnant have the right to an abortion? What he doesn t dig into is that the right to an abortion is really just an extension of bodily autonomy turning one facet of bodily autonomy into the legal right to have a medical procedure. I think this is worth thinking about more turning high level human rights into the mundane rights, and acknowledging that not everyone can or needs them.

Norbert Preining: KDE/Plasma Status Update 2020-09-03

Yesterday I have updated my builds of Plasma for Debian to Plasma 5.19.5, which are now available from the usual sources, nothing has changed. [Update 2020-09-03: KDE Apps 20.08.1 are also now available] On a different front, there are good news concerning updates in Debian proper: Together with Scarlett Moore and Patrick Franz we are in the process of updating the official Debian packages. The first bunch of packages has been uploaded to experimental, and after NEW processing the next group will go there, too. This is still 5.19.4, but a great step forward. I expect that all of Plasma 5.19.4 will be available in experimental in the next weeks, and soon after also in Debian/unstable. Again, thanks to Scarlett and Patrick for the good collaboration, this is very much appreciated!

2 September 2020

Kees Cook: security things in Linux v5.6

Previously: v5.5. Linux v5.6 was released back in March. Here s my quick summary of various features that caught my attention: WireGuard
The widely used WireGuard VPN has been out-of-tree for a very long time. After 3 1/2 years since its initial upstream RFC, Ard Biesheuvel and Jason Donenfeld finished the work getting all the crypto prerequisites sorted out for the v5.5 kernel. For this release, Jason has gotten WireGuard itself landed. It was a twisty road, and I m grateful to everyone involved for sticking it out and navigating the compromises and alternative solutions. openat2() syscall and RESOLVE_* flags
Aleksa Sarai has added a number of important path resolution scoping options to the kernel s open() handling, covering things like not walking above a specific point in a path hierarchy (RESOLVE_BENEATH), disabling the resolution of various magic links (RESOLVE_NO_MAGICLINKS) in procfs (e.g. /proc/$pid/exe) and other pseudo-filesystems, and treating a given lookup as happening relative to a different root directory (as if it were in a chroot, RESOLVE_IN_ROOT). As part of this, it became clear that there wasn t a way to correctly extend the existing openat() syscall, so he added openat2() (which is a good example of the efforts being made to codify Extensible Syscall arguments). The RESOLVE_* set of flags also cover prior behaviors like RESOLVE_NO_XDEV and RESOLVE_NO_SYMLINKS. pidfd_getfd() syscall
In the continuing growth of the much-needed pidfd APIs, Sargun Dhillon has added the pidfd_getfd() syscall which is a way to gain access to file descriptors of a process in a race-less way (or when /proc is not mounted). Before, it wasn t always possible make sure that opening file descriptors via /proc/$pid/fd/$N was actually going to be associated with the correct PID. Much more detail about this has been written up at LWN. openat() via io_uring
With my attack surface reduction hat on, I remain personally suspicious of the io_uring() family of APIs, but I can t deny their utility for certain kinds of workloads. Being able to pipeline reads and writes without the overhead of actually making syscalls is pretty great for performance. Jens Axboe has added the IORING_OP_OPENAT command so that existing io_urings can open files to be added on the fly to the mapping of available read/write targets of a given io_uring. While LSMs are still happily able to intercept these actions, I remain wary of the growing syscall multiplexer that io_uring is becoming. I am, of course, glad to see that it has a comprehensive (if out of tree ) test suite as part of liburing. removal of blocking random pool
After making algorithmic changes to obviate separate entropy pools for random numbers, Andy Lutomirski removed the blocking random pool. This simplifies the kernel pRNG code significantly without compromising the userspace interfaces designed to fetch cryptographically secure random numbers. To quote Andy, This series should not break any existing programs. /dev/urandom is unchanged. /dev/random will still block just after booting, but it will block less than it used to. See LWN for more details on the history and discussion of the series. arm64 support for on-chip RNG
Mark Brown added support for the future ARMv8.5 s RNG (SYS_RNDR_EL0), which is, from the kernel s perspective, similar to x86 s RDRAND instruction. This will provide a bootloader-independent way to add entropy to the kernel s pRNG for early boot randomness (e.g. stack canary values, memory ASLR offsets, etc). Until folks are running on ARMv8.5 systems, they can continue to depend on the bootloader for randomness (via the UEFI RNG interface) on arm64. arm64 E0PD
Mark Brown added support for the future ARMv8.5 s E0PD feature (TCR_E0PD1), which causes all memory accesses from userspace into kernel space to fault in constant time. This is an attempt to remove any possible timing side-channel signals when probing kernel memory layout from userspace, as an alternative way to protect against Meltdown-style attacks. The expectation is that E0PD would be used instead of the more expensive Kernel Page Table Isolation (KPTI) features on arm64. powerpc32 VMAP_STACK
Christophe Leroy added VMAP_STACK support to powerpc32, joining x86, arm64, and s390. This helps protect against the various classes of attacks that depend on exhausting the kernel stack in order to collide with neighboring kernel stacks. (Another common target, the sensitive thread_info, had already been moved away from the bottom of the stack by Christophe Leroy in Linux v5.1.) generic Page Table dumping
Related to RISCV s work to add page table dumping (via /sys/fs/debug/kernel_page_tables), Steven Price extracted the existing implementations from multiple architectures and created a common page table dumping framework (and then refactored all the other architectures to use it). I m delighted to have this because I still remember when not having a working page table dumper for ARM delayed me for a while when trying to implement upstream kernel memory protections there. Anything that makes it easier for architectures to get their kernel memory protection working correctly makes me happy. That s in for now; let me know if there s anything you think I missed. Next up: Linux v5.7.

2020, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 License.
CC BY-SA 4.0

Vincent Bernat: Syncing MySQL tables with a custom Ansible module

The community.mysql collection from Ansible Galaxy provides a mysql_query module to run arbitrary MySQL queries. Unfortunately, it does not support check mode nor the --diff flag. It is also unable to tell if there was a change. Let s write a specific Ansible module to workaround these issues.

Notice I recommend that you read Writing a custom Ansible module as an introduction.

Code The module has the following signature and it executes the provided SQL statements in a single transaction. It needs a list of the affected tables to be able to detect and show the changes. 
mysql_sync:
  sql:  
    DELETE FROM rules WHERE name LIKE 'CMDB:%';
    INSERT INTO rules (name, rule) VALUES
      ('CMDB: check for cats', ':is(object, "CAT")'),
      ('CMDB: check for dogs', ':is(object, "DOG")');
    REPLACE INTO webhooks (name, url) VALUES
      ('OpsGenie', 'https://opsgenie/something/token'),
      ('Slack', 'https://slack/something/token');
  user: monitoring
  password: Yooghah5
  database: monitoring
  tables:
    - rules
    - webhooks

Prerequisites The module does not enforce idempotency, but it is expected you provide appropriate SQL queries. In the above example, idempotency is achieved because the content of the rules table is deleted and recreated from scratch while the rows in the webhooks table are replaced if they already exist. You need the PyMySQL package.

Module definition Starting from the skeleton described in the previous article, here is the module definition: 
module_args = dict(
    sql=dict(type='str', required=True),
    user=dict(type='str', required=True),
    password=dict(type='str', required=True, no_log=True),
    database=dict(type='str', required=True),
    tables=dict(type='list', required=True, elements='str'),
)
result = dict(
    changed=False
)
module = AnsibleModule(
    argument_spec=module_args,
    supports_check_mode=True
)
The password is marked with no_log to ensure it won t be displayed or stored, notably when ansible-playbook runs in verbose mode. There is no host option as the module is executed on the MySQL host. Strong authentication using certificates is not implemented either. This matches our goal with custom modules: only implement what you strictly need.

Getting the current rows The next step is to retrieve the records currently in the database. The got dictionary is a mapping from table names to the list of rows they contain: 
got =  
tables = module.params['tables']
connection = pymysql.connect(
    user=module.params['user'],
    password=module.params['password'],
    db=module.params['database'],
    charset='utf8mb4',
    cursorclass=pymysql.cursors.DictCursor
)
with connection.cursor() as cursor:
    for table in tables:
        cursor.execute("SELECT * FROM  ".format(table))
        got[table] = cursor.fetchall()

Computing the changes Let s now build the wanted dictionary. The trick is to execute the SQL statements in a transaction without issuing a final commit. The changes will be invisible1 to other readers and we can compare the final rows with the rows collected in got: 
wanted =  
sql = module.params['sql']
statements = [statement.strip()
              for statement in sql.split(";\n")
              if statement.strip()]
with connection.cursor() as cursor:
    for statement in statements:
        try:
            cursor.execute(statement)
        except pymysql.OperationalError as err:
            code, message = err.args
            result['msg'] = "MySQL error for  :  ".format(
                statement,
                message)
            module.fail_json(**result)
    for table in tables:
        cursor.execute("SELECT * FROM  ".format(table))
        wanted[table] = cursor.fetchall()
The first for loop executes each statement. On error, we return a helpful message containing the faulty one. The second for loop records the final rows of each table in wanted.

Applying changes Back to the skeleton described in the previous article, the last step is to apply the changes if there is a difference between got and wanted when not running with check mode. The diff object is a bit more elaborate as it is built table by table. This enables Ansible to display the name of each table before the diff representation: 
if got != wanted:
    result['changed'] = True
    result['diff'] = [dict(
        before_header=table,
        after_header=table,
        before=yaml.safe_dump(got[table]),
        after=yaml.safe_dump(wanted[table]))
                      for table in tables
                      if got[table] != wanted[table]]
if module.check_mode or not result['changed']:
    module.exit_json(**result)
Applying the changes is quite trivial: just commit them! Otherwise, they are lost when the module exits. 
connection.commit()
The complete code is available on GitHub. Compared to the mysql_query module, this one supports the check mode, signals correctly if there is a change and displays the differences. However, it should not be used with huge tables, as it would try to load them in memory.
  1. The tables need to use the InnoDB storage engine. Moreover, MySQL does not know how to use transactions with DDL statements: do not modify table definitions!

Vincent Bernat: Syncing SSH keys on Cisco IOS-XR with a custom Ansible module

The cisco.iosxr collection from Ansible Galaxy provides an iosxr_user module to manage local users, along with their SSH keys. However, the module is quite slow, do not display a diff for changed SSH keys, never signal change when a key is modified, and does not delete obsolete keys. Let s write a custom Ansible module managing only the SSH keys while fixing these issues.

Notice I recommend that you read Writing a custom Ansible module as an introduction.

How to add an SSH key to a user Adding SSH keys to users in Cisco IOS-XR is quite undocumented. First, you need to encode the key with the ssh-rsa key ASN.1 format, like an OpenSSH public key, but without the base64-encoding: 
$ awk ' print $2 ' id_rsa.pub \
      base64 -d \
    > publickey_vincent.raw
Then, you upload the key with SCP to harddisk:/publickey_vincent.raw and import it for the current user with the following IOS command: 
crypto key import authentication rsa harddisk:/publickey_vincent.b64
However, if you want to import a key for another user, you need to be part of the root-system group: 
username vincent
 group root-lr
 group root-system
With the following admin command, you can attach a key to another user: 
admin crypto key import authentication rsa username cedric harddisk:/publickey_cedric.b64

Code The module has the following signature and it installs the specified key for each user and remove keys from retired users the ones we do not specify. 
iosxr_users:
  keys:
    vincent: ssh-rsa AAAAB3NzaC1yc2EAA[ ]ymh+YrVWLZMJR
    cedric:  ssh-rsa AAAAB3NzaC1yc2EAA[ ]RShPA8w/8eC0n

Prerequisites Unlike the iosxr_user module, our custom module only handles SSH keys, one per user. Therefore, the user definitions have to already exist in the running configuration.1 Moreover, the user defined in ansible_user needs to be in the root-system group. The cisco.iosxr collection must also be installed as the module relies on its code. When running the module, ansible_connection needs to be set to network_cli and ansible_network_os to iosxr. These variables are usually defined in the inventory.

Module definition Starting from the skeleton described in the previous article, we define the module: 
module_args = dict(
    keys=dict(type='dict', elements='str', required=True),
)
module = AnsibleModule(
    argument_spec=module_args,
    supports_check_mode=True
)
result = dict(
    changed=False
)

Getting the installed keys The next step is to retrieve the keys currently installed. This can be done with the following command: 
# show crypto key authentication rsa all
Key label: vincent
Type     : RSA public key authentication
Size     : 2048
Imported : 16:17:08 UTC Tue Aug 11 2020
Data     :
 30820122 300D0609 2A864886 F70D0101 01050003 82010F00 3082010A 02820101
 00D81E5B A73D82F3 77B1E4B5 949FB245 60FB9167 7CD03AB7 ADDE7AFE A0B83174
 A33EC0E6 1C887E02 2338367A 8A1DB0CE 0C3FBC51 15723AEB 07F301A4 B1A9961A
 2D00DBBD 2ABFC831 B0B25932 05B3BC30 B9514EA1 3DC22CBD DDCA6F02 026DBBB6
 EE3CFADA AFA86F52 CAE7620D 17C3582B 4422D24F D68698A5 52ED1E9E 8E41F062
 7DE81015 F33AD486 C14D0BB1 68C65259 F9FD8A37 8DE52ED0 7B36E005 8C58516B
 7EA6C29A EEE0833B 42714618 50B3FFAC 15DBE3EF 8DA5D337 68DAECB9 904DE520
 2D627CEA 67E6434F E974CF6D 952AB2AB F074FBA3 3FB9B9CC A0CD0ADC 6E0CDB2A
 6A1CFEBA E97AF5A9 1FE41F6C 92E1F522 673E1A5F 69C68E11 4A13C0F3 0FFC782D
 27020301 0001
[ ]
ansible_collections.cisco.iosxr.plugins.module_utils.network.iosxr.iosxr contains a run_commands() function we can use: 
command = "show crypto key authentication rsa all"
out = run_commands(module, command)
out = out[0].replace(' \n', '\n')
A common library to parse a command output is textfsm: a Python module using a template-based state machine for parsing semi-formatted text. 
template = r"""
Value Required Label (\w+)
Value Required,List Data ([A-F0-9 ]+)
Start
 ^Key label: $ Label 
 ^Data\s+: -> GetData
GetData
 ^ $ Data 
 ^$$ -> Record Start
""".lstrip()
re_table = textfsm.TextFSM(io.StringIO(template))
got =  data[0]: "".join(data[1]).replace(' ', '')
       for data in re_table.ParseText(out)
got is a dictionary associating key labels, considered as usernames, with a hexadecimal representation of the public key currently installed. It looks like this: 
>>> pprint(got)
 'alfred': '30820122300D0609[ ]6F0203010001',
 'cedric': '30820122300D0609[ ]710203010001',
 'vincent': '30820122300D0609[ ]270203010001'

Comparing with the wanted keys Let s now build the wanted dictionary using the same structure. In module.params['keys'], we have a dictionary associating usernames to public SSH keys in the OpenSSH format: 
>>> pprint(module.params['keys'])
 'cedric': 'ssh-rsa AAAAB3NzaC1yc2[ ]',
 'vincent': 'ssh-rsa AAAAB3NzaC1yc2[ ]'
We need to convert these keys in the same hexadecimal representation used by Cisco above. The ssh-keygen command and some glue can do the conversion:2 
$ ssh-keygen -f id_rsa.pub -e -mPKCS8 \
     grep -v '^---' \
     base64 -d \
     hexdump -e '4/1 "%0.2X"'
30820122300D06092[ ]782D270203010001
Assuming we have a ssh2cisco() function doing that, we can build the wanted dictionary: 
wanted =  k: ssh2cisco(v)
          for k, v in module.params['keys'].items()

Applying changes Back to the skeleton described in the previous article, the last step is to apply the changes if there is a difference between got and wanted when not running with check mode. The part comparing got and wanted is taken verbatim from the skeleton module: 
if got != wanted:
    result['changed'] = True
    result['diff'] = dict(
        before=yaml.safe_dump(got),
        after=yaml.safe_dump(wanted)
    )
if module.check_mode or not result['changed']:
    module.exit_json(**result)
Let s copy the new or changed keys and attach them to their respective users. For this purpose, we reuse the get_connection() and copy_file() functions from ansible_collections.cisco.iosxr.plugins.module_utils.network.iosxr.iosxr. 
conn = get_connection(module)
for user in wanted:
    if user not in got or wanted[user] != got[user]:
        dst = f"/harddisk:/publickey_ user .raw"
        with tempfile.NamedTemporaryFile() as src:
            decoded = base64.b64decode(
                module.params['keys'][user].split()[1])
            src.write(decoded)
            src.flush()
            copy_file(module, src.name, dst)
    command = ("admin crypto key import authentication rsa "
               f"username  user   dst ")
    conn.send_command(command, prompt="yes/no", answer="yes")
Then, we remove obsolete keys: 
for user in got:
    if user not in wanted:
        command = ("admin crypto key zeroize authentication rsa "
                   f"username  user ")
        conn.send_command(command, prompt="yes/no", answer="yes")
The complete code is available on GitHub. Compared to the iosxr_user module, this one displays a diff when running with --diff, correctly signals a change, is faster, 3 and deletes unwanted SSH keys. However, it is unable to create users and cannot configure passwords or multiple SSH keys.
  1. In our environment, the Ansible playbook pushes a full configuration, including the user definitions. Then, it synchronizes the SSH keys.
  2. Despite the argument provided to ssh-keygen, the format used by Cisco is not PKCS#8. This is the ASN.1 representation of a Subject Public Key Info structure, as defined in RFC 2459. Moreover, PKCS#8 is a format for a private key, not a public one.
  3. The main factors for being faster are:
    • not creating users, and
    • not reuploading existing SSH keys.

Vincent Bernat: Writing a custom Ansible module

Ansible ships a lot of modules you can combine for your configuration management needs. However, the quality of these modules may vary widely. Sometimes, it may be quicker and more robust to write your own module instead of shopping and assembling existing ones.1 In my opinion, a robust module exhibits the following characteristics: In a nutshell, it means the module can run with --diff --check and shows the changes it would apply. When run twice in a row, the second run won t apply or signal changes. The last bullet point suggests the module should be able to delete outdated objects configured during previous runs.2 The module code should be minimal and tailored to your needs. Making the module generic for use by other users is a non-goal. Less code usually means less bugs and easier to understand. I do not cover testing here. It is undeniably a good practice, but it requires a significant effort. In my opinion, it is preferable to have a well written module matching the above characteristics rather than a module that is well tested but without them or a module requiring further (untested) assembly to meet your needs.

Module skeleton Ansible documentation contains instructions to build a module, along with some best practices. As one of our non-goal is to distribute it, we choose to take some shortcuts and skip some of the boilerplate. Let s assume we build a module with the following signature: 
custom_module:
  user: someone
  password: something
  data: "some random string"
There are various locations you can put a module in Ansible. A common possibility is to include it into a role. In a library/ subdirectory, create an empty __init__.py file and a custom_module.py file with the following code:3 
#!/usr/bin/python
import yaml
from ansible.module_utils.basic import AnsibleModule
def main():
    # Define options accepted by the module.  
    module_args = dict(
        user=dict(type='str', required=True),
        password=dict(type='str', required=True, no_log=True),
        data=dict(type='str', required=True),
    )
    module = AnsibleModule(
        argument_spec=module_args,
        supports_check_mode=True
    )
    result = dict(
        changed=False
    )
    got =  
    wanted =  
    # Populate both  got  and  wanted .  
    # [...]
    if got != wanted:
        result['changed'] = True
        result['diff'] = dict(
            before=yaml.safe_dump(got),
            after=yaml.safe_dump(wanted)
        )
    if module.check_mode or not result['changed']:
        module.exit_json(**result)
    # Apply changes.  
    # [...]
    module.exit_json(**result)
if __name__ == '__main__':
    main()
The first part, in , defines the module, with the accepted options. Refer to the documentation on argument_spec for more details. The second part, in , builds the got and wanted variables. got is the current state while wanted is the target state. For example, if you need to modify records in a database server, got would be the current rows while wanted would be the modified rows. Then, we compare got and wanted. If there is a difference, changed is switched to True and we prepare the diff object. Ansible uses it to display the differences between the states. If we are running in check mode or if no change is detected, we stop here. The last part, in , applies the changes. Usually, it means iterating over the two structures to detect the differences and create the missing items, delete the unwanted ones and update the existing ones.

Documentation Ansible provides a fairly complete page on how to document a module. I advise you to take a more minimal approach by only documenting each option sparingly,4 skipping the examples and only documenting return values if it needs to. I usually limit myself to something like this: 
DOCUMENTATION = """
---
module: custom_module.py
short_description: Pass provided data to remote service
description:
  - Mention anything useful for your workmate.
  - Also mention anything you want to remember in 6 months.
options:
  user:
    description:
      - user to identify to remote service
  password:
    description:
      - password for authentication to remote service
  data:
    description:
      - data to send to remote service
"""

Error handling If you run into an error, you can stop the execution with module.fail_json(): 
module.fail_json(
    msg=f"remote service answered with  code :  message ",
    **result
)
There is no requirement to intercept all errors. Sometimes, not swallowing an exception provides better information than replacing it with a generic message.

Returning additional values A module may return additional information that can be captured to be used in another task through the register directive. For this purpose, you can add arbitrary fields to the result dictionary. Have a look at the documentation for common return values. You should try to add these fields before exiting the module when in check mode. The returned values can be documented.

Examples Here are several examples of custom modules following the previous skeleton. Each example highlight why a custom module was written instead of assembling existing modules.
  1. Also, when using modules from Ansible Galaxy, you introduce a dependency to a third-party. This is not something that should be decided lightly: it may break later, it may only meet 80% of the needs, it may add bugs.
  2. Some declarative systems, like Terraform, exhibits all these behaviors.
  3. Do not worry about the shebang. It is hardcoded to /usr/bin/python. Ansible will modify it to match the chosen interpreter on the remote host. You can write Python 3 code if ansible_python_interpreter evaluates to a Python 3 interpreter.
  4. The main issue I have with this non-programmatic approach to documentation is that it partly repeats the information contained in argument_spec. I think an auto-documenting structure would avoid this.

Elana Hashman: My term at the Open Source Initiative thus far

When I ran for the OSI board in early 2019, I set three goals for myself: Now that the OSI has announced hiring an interim General Manager, I thought it would be a good time to publicly reflect on what I've accomplished and what I'd like to see next. As I promised in my campaign pitch, I aim to be publicly accountable :) Growing the OSI's membership I have served as our Membership Committee Chair since the May 2019 board meeting, tasked with devising and supervising strategy to increase membership and deliver value to members. As part of my election campaign last year, I signed up over 50 new individual members. Since May 2019, we've seen strong 33% growth of individual members, to reach a new all-time high over 600 (638 when I last checked). I see the OSI as a relatively neutral organization that occupies a unique position to build bridges among organizations within the FOSS ecosystem. In order to facilitate this, we need a representative membership, and we need to engage those members and provide forums for cross-pollination. As Membership Committee Chair, I have been running quarterly video calls on Jitsi for our affiliate members, where we can share updates between many global organizations and discuss challenges we all face. But it's not enough just to hold the discussion; we also need to bring fresh new voices into the conversation. Since I've joined the board, I'm thrilled to say that 16 new affiliate members joined (in chronological order) for a total of 81: I was also excited to run a survey of the OSI's individual and affiliate membership to help inform the future of the organization that received 58 long-form responses. The survey has been accepted by the board at our August meeting and should be released publicly soon! Defending the Open Source Definition When I joined the board, the first committee I joined was the License Committee, which is responsible for running the licence review process, making recommendations on new licenses, and maintaining our existing licenses. Over the past year, under Pamela Chestek's leadership as Chair, the full board has approved the following licenses (with SPDX identifiers in brackets) on the recommendation of the License Committee: We withheld approval of the following licenses: I've also worked to define the scope of work for hiring someone to improve our license review process, which we have an open RFP for! Chopping wood and carrying water I joined the OSI with the goal of improving an organization I didn't think was performing up to its potential. Its membership and board were not representative of the wider open source community, its messaging felt outdated, and it seemed to be failing to rise to today's challenges for FOSS. But before one can rise to meet these challenges, you need a strong foundation. The OSI needed the organizational structure, health, and governance in order to address such questions. Completing that work is essential, but not exactly glamourous and it's a place that I thrive. Honestly, I don't (yet?) want to be the public face of the organization, and I apologize to those who've missed me at events like FOSDEM. I want to talk a little about some of my behind-the-scenes activities that I've completed as part of my board service: All of this work is intended to improve the organization's health and provide it with an excellent foundation for its mission. Defining the future of open source Soon after I was elected to the board, I gave a talk at Brooklyn.js entitled "The Future of Open Source." In this presentation, I pondered about the history and future of the free and open source software movement, and the ethical questions we must face. In my election campaign, I wrote "Software licenses are a means, not an end, to open source software. Focusing on licensing is necessary but not sufficient to ensure a vibrant, thriving open source community. Focus on licensing to the exclusion of other serious community concerns is to our collective detriment." My primary goal for my first term on the board was to ensure the OSI would be positioned to answer wider questions about the open source community and its future beyond licenses. Over the past two months, I supported Megan Byrd-Sanicki's suggestion to hold (and then participated in, with the rest of the board) organizational strategy sessions to facilitate our long-term planning. My contribution to help inform these sessions was providing the member survey on behalf of the Membership Committee. Now, I think we are much better equiped to face the hard questions we'll have to tackle. In my opinion, the Open Source Initiative is better positioned than ever to answer them, and I can't wait to see what the future brings. Hope to see you at our first State of the Source conference next week!

Next.

Previous.