Search Results: "hartmans"

28 April 2021

Russell Coker: Links April 2021

Dr Justin Lehmiller s blog post comparing his official (academic style) and real biographies is interesting [1]. Also the rest of his blog is interesting too, he works at the Kinsey Institute so you know he s good. Media Matters has an interesting article on the spread of vaccine misinformation on Instagram [2]. John Goerzen wrote a long post summarising some of the many ways of having a decentralised Internet [3]. One problem he didn t address is how to choose between them, I could spend months of work to setup a fraction of those services. Erasmo Acosta wrote an interesting medium article Could Something as Pedestrian as the Mitochondria Unlock the Mystery of the Great Silence? [4]. I don t know enough about biology to determine how plausible this is. But it is a worry, I hope that humans will meet extra-terrestrial intelligences at some future time. Meredith Haggerty wrote an insightful Medium article about the love vs money aspects of romantic comedies [5]. Changes in viewer demographics would be one factor that makes lead actors in romantic movies significantly less wealthy in recent times. Informative article about ZIP compression and the history of compression in general [6]. Vice has an insightful article about one way of taking over SMS access of phones without affecting voice call or data access [7]. With this method the victom won t notice that they are having their sservice interfered with until it s way too late. They also explain the chain of problems in the US telecommunications industry that led to this. I wonder what s happening in this regard in other parts of the world. The clown code of ethics (8 Commandments) is interesting [8]. Sam Hartman wrote an insightful blog post about the problems with RMS and how to deal with him [9]. Also Sam Whitton has an interesting take on this [10]. Another insightful post is by Selam G about RMS long history of bad behavior and the way universities are run [11]. Cory Doctorow wrote an insightful article for Locus about free markets with a focus on DRM on audio books [12]. We need legislative changes to fix this!

24 March 2021

Sam Hartman: Making our Community Safe: the FSF and rms

I felt disgust and horror when I learned yesterday that rms had returned to the FSF board. When rms resigned back in September of 2019, I was Debian Project Leader. At that time, I felt two things. First, I was happy that the community was finally taking a stand in favor of inclusion, respect, and creating a safe, welcoming place to do our work. It was long past time for rms to move on. But I also felt thankful that rms was not my problem to solve. In significant part because of rms, I had never personally been that involved in the FSF. I considered drafting a statement as Debian Project Leader. I could have talked about how through our Diversity Statement and Code of Conduct we had taken a stand in favor of inclusion and respect. I could have talked about how rms's actions displayed a lack of understanding and empathy and how this created a community that was neither welcoming nor respectful. I didn't. I guess I didn't want to deal with confirming I had sufficient support in the project. I wanted to focus on internal goals, and I was healing and learning from some mistakes I made earlier in the year. It looked like other people were saying what needed to be said and my voice was not required. Silence was a mistake.

It's a mistake I've been making all throughout my interactions with rms. Enough is enough. It's long past time I added my voice to those who cry for accountability and who will not sit aside while rms's disrespect and harm is tolerated.

The first time I was silent about rms was around 15 years ago. I was at a science fiction convention in a crowded party. I didn't know anyone, other than the host of the party. I was out of my depth. I heard his voice---I recognized it from the Share the Software Song. He was hitting on some girl, talking about how he invented Emacs. As best I could tell, she didn't even know what Emacs was. Back then, I wondered what she saw in the interaction; why she stuck around even though she didn't know what he was talking about. I sure didn't want to be around; the interaction between the two of them was making me uncomfortable. Besides, the wings on her costume kept hitting me in the face. So I left as fast as I could.

I've learned a lot about creating safe spaces and avoiding sexual harassment since then. Thinking back, she was probably hitting me because she was trying to back away and getting crowded. If this happened today, I think I would do a better job of owning my responsibility for helping keep the space around me safe. I've learned better techniques for checking in to make sure people around me are comfortable.

I didn't come to silence alone: I had been educated into the culture of avoiding rms and not calling him out. There was a running game in the group of computer security professionals I learned from. The goal was to see how much you could contribute to free software and computer security without being recognized by or interacting with rms. And so, indoctrinated into a culture of silence about the harm that rms caused, I took my first step.

Things weren't much better when I attended Libreplanet 2019 just before taking office as Debian Project Leader. I had stayed away from the conference in large part because of rms. But there were Debian people there, and I was missing community interaction. Unfortunately, I saw that even after the problems of 2018, rms was still treating himself as above community standards. He interrupted speakers, objecting to how they phrased the problem they were considering. After a speech on codes of conduct in the free software community, he cornered the talk organizer to "ask her opinion" about the GNU project's lack of code on conduct. He wasn't asking for an opinion. He was justifying himself; there wasn't much listening in the conversation I heard. Aspects of that conversation crossed professional boundaries for what should be said. The talk organizer was okay--we talked about it after--but if we did a better job of policing our community, that wouldn't even be a question. I think the most telling sign was a discussion with an FSF board member. We were having a great conversation, but he had to interrupt it. He was on rms duty (my words) at the next session. The board had decided it was necessary to have members there so that the staff would not be put in awkward positions by their president. If someone needed to call rms out, it could be a board member rather than the staff members of the conduct team.

And yet again, I held my silence. It's so easy to keep silent. It's not that I never speak up. There are communities where I have called people out. But it's hard to paint that target on yourself. It's hard to engage and to stand strong for a community's standards when you aren't the target. It's hard to approach these problems while maintaining empathy for everyone involved. Some people give into the rage; I don't have that option if I want to be the person I've chosen to be. And so, when I do speak up, the emotional cost is high.

Yet, it's long past time I raised my voice on this issue. Rms has demonstrated that he cannot hold to standards of respect for others, respect for their boundaries, or standards of community safety. We need those standards to be a welcoming community.

If the people who came before me--those who taught me the game of avoiding rms--had spoken up, the community could have healed before I even came on the scene. If I and others had stood up fifteen years ago, we'd have another couple generations who were more used to respect, inclusion, welcoming and safety. The FSF board could have done their job back in 2018. And perhaps if more of us had spoken out in 2019, the FSF board would have found the strength to stand strong and not accept rms's return.

And so, finally, I raise my voice. I signed the open letter calling for the resignation of rms and the entire FSF board. Perhaps if we all get used to raising our voice, it will be easier. Perhaps if we stand together, taking the path of community rather than the path of silence, we'll have the support we need to create communities inclusive enough to welcome everyone who can contribute. For me, I'm done being silent.

There's one criticism of the open letter I'd like to respond to. I've heard concerns about asking for the resignation of the entire FSF board under the understanding that some board members voted against rms's return. It should be obvious why those who voted for rms's return need to resign. But resignation does not always mean you did something wrong. If you find yourself in a leadership role in an organization that takes decisions in significant conflict with your standards of ethics, resignation is also the right path. Staying on the board even if you voted against rms's return means that you consider voting for rms to be a reasonable thing to do. It means that even if you disagreed with it, you can still be part of an organization that takes the path of welcoming rms. At this point, I cannot do that, nor can I support leaders in the FSF who do.

6 August 2020

Sam Hartman: Good Job Debian: Compatibility back to 1999

So, I needed a container of Debian Slink (2.1), released back in 1999. I expected this was going to be a long and involved process. Things didn't look good from the start:
root@mount-peerless:/usr/lib/python3/dist-packages/sqlalchemy# debootstrap  slink /build/slink2 
http://archive.debian.org/debian                                                               
E: No such script: /usr/share/debootstrap/scripts/slink

Hmm, I thought I remembered slink support for debootstrap--not that slink used debootstrap by default--back when I was looking through the debootstrap sources years ago. Sure enough looking through the changelogs, slink support was dropped back in 2005.
Okay, well, this isn't going to work either, but I guess I could try debootstrapping sarge and from there go back to slink.
Except it worked fine.
Go us!

9 March 2020

Sam Hartman: Forged Email

Last night, a series of forged emails was sent to a number of places around the Debian, Ubuntu and Free Software communities. The meat of the mail was a fake message from me to debian-private with the subject "DebConf19 Diversity Girls." I didn't write such a message.
I view this message as the latest installment in a campaign of attacks on Debian that attempt to undermine the project and take up the time of our members.
I was expecting something like this: yesterday, I banned Daniel Pocock from the project.There's been a pattern of related events over the past year and a half:

27 October 2017

Sam Hartman: How Can Debian Turn Disagreement into Something that Makes us Stronger

Recently, when asked to engage with the Debian Technical Committee, a maintainer chose to orphan their package rather than discuss the issue brought before the committee. In another decision earlier this year, a maintainer orphaned their package indicating a lack of respect for the approach being taken and the process. Unfortunately, this joins an ever longer set of issues where people walk away from the TC process disheartened and upset.

For me personally the situations where maintainers walked away from the process were hard. People I respect and admire were telling me that they were unwilling to participate in our dispute resolution process. In one case the maintainer explicitly did not respect a process I had been heavily involved in. As someone who values understanding and build a team, I feel disappointed and hurt thinking about this.

Unfortunately, I don't feel much better when I consider several of the other issues brought before the TC. In a number of cases, the process has concluded, but it feels like a pyrrhic victory. We have an answer, but often we never reached an understanding of one of the key stakeholder's positions. People are sufficiently disappointed or frustrated that they reduce their involvement. That is, the answer is not one they can live with. Sometimes, I'm not even sure that answering the question was worth the cost.

My initial suspicion as I begin to consider issues that have come before the TC in the last few years is that as a necessary consequence of how the TC operates, we will drive a significant chunk of those who come to us away from our community. I will not be part of that. If I end up eventually agreeing with this suspicion, I will either work to improve the TC process or find a different way to contribute to the project that is more aligned with my goals.


Our Community is More Important than Technical Correctness I firmly believe that Debian's strength is its community. Distributions, upstreams, free-software advocates, corporate interests, and everything in between work together. Because we are working on a concrete product,we actually have to understand how our needs conflict and compliment each other. Also, like any organization, our ability to get things done is dependent on our contributors and the time we all put in.

I cannot think of anything more harmful we can do than drive away a constructive contributor. Technical quality is important: many of us will eventually go away if quality drops too much. However, Debian itself can be improved incrementally. Yes, new people do join Debian. However, it takes a lot of new people to make up for a situation where everyone involved is frustrated, and some leave and tell their friends to stay away from Debian.

When a Disagreement Reaches the TC When a disagreement reaches the TC, we know a few things. First, people have not have not been able to resolve it on their own. Second, the disagreement is important enough to at least one of the parties to ask for outside help.

If the TC were to simply announce a decision, it is very likely that at least one party would feel frustrated and disappointed. If the decision is important to someone it almost always means that they care about the outcome. If previous efforts have not reached agreement then there is an outcome that is undesired. While it's possible under the constitution that two parties could bring an issue to the TC mutually where the dynamics could be different, this does not happen in practice.

When we "lose" When a decision making process decides to select one of my undesired outcomes, I have strong negative feelings. First, there is the technical result itself. Because of an adverse decision it is generally harder for me to do my technical work, or some ethical position or group I care about is disadvantaged. However, the strongest feelings are related to needs about my place in the community, not a direct response to the technical decision. I worry about whether issues I care about will be valued in the future; I would likely feel weary or scared thinking of contributing in an environment where issues that matter to me aren't going to be considered. I tend to feel frustrated if my position was not adequately considered this time around.

Often, the strongest feelings do not stem from how I am impacted by the decision itself. Thus, if my more general needs are addressed, I will feel a lot better about not having my preferred outcome selected. Confidence that my position was understood is the single biggest factor in being able to accept the result. If the community took the time to understand me, then I have confidence that I am valued. I can advocate for change and be part of the ongoing discussion. If I understand the other positions then I can understand that the position was not arbitrary. Understanding the other positions is also a prerequisite to seeing how things I value were considered, especially when the tradeoff did not ultimately favor these values.

My conversations with others about their experience with conflict suggest my feelings and needs are common.


However, the TC focuses almost exclusively on the technical matter before it. I don't think the TC has done a good job of actually understanding maintainers or the people bringing issues to us. Especially when we're fairly sure we understand what the ultimate decision will be, we focus on getting to that decision. Of course part of actually understanding someone is considering what they have to say. Even when you have high confidence in an outcome, if you want someone to feel understood, you do have to listen at a point where you can consider what they bring forward.

Frustration of Being Questioned

On the other side, it's frustrating to have your decisions questioned. Reviewing a decision takes time. Even if the TC agrees with a maintainer, asking that maintainer to sit through a long process can create frustrations as deep as any I discussed in the previous section.

So the process is doubly bad for maintainers. Someone can bring up an issue which requires precious time. Then if the TC decides against the maintainer, we have all the problems I discussed previously.

I acknowledge this. A good process must respect the maintainer's time. However, it must respect the community members who disagree with maintainers. Everyone who brings an issue to the TC is a developer. They have contributed significantly to our community and are part of our team. Yes, we need to protect the process against abuse. But in a very real sense if we aren't willing to consider an issue and we aren't willing to engage with someone, understand why they think the issue should be considered, and work until they understand why we made our decision, we are saying we do not respect them enough to take that time. We should expect that to drive people away.

I wonder if the solution here involves a two phase process. During the first phase we work with someone bringing an issue to us until we understand it. We either engage the maintainer at that point, spend the time to help the person bringing an issue to us understand why we are not engaging the maintainer, or have decided the issue is abusive of the processs/misdirected. For this to work maintainers would have to trust us enough to actually be willing to sit back and not spend a lot of their time.

Sam, It's Just Personalities

One criticism I've heard as I discuss this is that a lot of the negative feelings surround interpersonal conflict and are personality conflicts. Yes, personalities matter. Yes, we re still healing from the Systemd discussion.

However, as a community, I think we need to figure out how we respect the inevitable personality conflicts. I can think of one or two developers I'm still upset with years after an issue. It happens. Perhaps if I were a maintainer when one of these developers raised a concern with my packages, I could ask that someone else be a primary advocate for an issue. Similarly, if a developer wanted to address an issue with me as a maintainer but would prefer not to deal with me, perhaps we could figure out some way that we could see if someone else shared the concern.

Dropping a Package is Sometimes an Answer

My concerns were sparked by instances where a maintainer dropped a package rather than interact with the TC. Sometimes that can be a healthy step in a transition. At Debconf this year, Enrico Zini gave a talk on consent culture in Debian. One of the key points of his talk is that we can find over time that what we're being asked to do in Debian is no longer consistent with our boundaries. If it isn t helping us move forward if it isn t fun then it is likely time to stop.

In that sense, approaching disagreement can be a great time to take stock and ask whether we still enjoy maintaining a package. If we don t, then stepping aside and letting others take it on can be a great way that we and they can be happier. I don t really think that s what happened in these instances though. Based on comments made to me, this sounded more like a lack of faith in being treated well or in our ability as a committee.

Even so, Enrico s talk applies in a number of ways here. He suggests that the approach we should take when someone is done and steps away is to thank then. I couldn t agree more. From the depth of my heart I offer thanks: Thank you for taking care of yourself and stepping away when it is no longer fun. Thank you for all the effort you have put into these packages. I hope to work with you on great things in the future.

12 August 2017

Sam Hartman: Debian: a Commons of Innovation

I recently returned from Debconf. This year at Debconf, Matthew Garrett gave a talk about the next twenty years in free software. In his talk he raised concerns that Debian might not be relevant in that ecosystem and talked about some of the trends that contribute to his concerns.
I was talking to Marga after the talk and she said that Debian used to be a lot more innovative than it is today.
My initial reaction was doubt; what she said didn't feel right to me. At the time I didn't have a good answer. Since then I've been pondering the issue, and I think I have a partial answer to both Marga and Matthew and so I'll share it here.
In the beginning Debian focused on a lot of technical innovations related to bringing an operating system together. We didn't understand how to approach builds and build dependencies in a uniform manner. Producing packages in a clean environment was hard. We didn't understand what we wanted out of packages in terms of a uniform approach to configuration handling and upgrades. To a large extent we've solved those problems.
However, as the community has grown, our interests are more diverse. Our users and free software (and the operating system we build together) are what bring us together: we still have a central focus. However, no one technical project captures us all. There's still significant technical innovation in the Debian ecosystem. That innovation happens in Debian teams, companies and organizations that interact with the Debian community. We saw several talks about such innovation this year. I found the talk about ostree and flatpak interesting, especially because it focused on people in the broader Debian ecosystem valuing Debian along with some of the same technologies that Matthew is worried will undermine our relevance.
Matthew talked about how Debian ends up being a man-in-the-middle. We're between users and developers. we're between distributions and upstreams. Users are frustrated because we hold back the latest version of software they want from getting to them. Developers are frustrated because we present our users with old versions of their software configured not as they would like, combined with different dependencies than they expect.
All these weaknesses are real.
However, I think Debian-in-the-middle is our greatest strength both on the technical and social front.
I value Debian because I get a relatively uniform interface to the software I use. I can take one approach to reporting bugs whether they are upstream or Debian specific. I expect the software to behave in uniform ways with regard to things covered by policy. I know that I'm not going to have to configure multiple different versions of core dependencies; for the most part system services are shared. When Debian has value it's because our users want those things we provide. Debian has also reviewed every source file in the software we ship to understand the license and license compatibility. As a free software supporter and as someone who consumes software in commercial context, that value alone is enormous.
The world has evolved and we're facing technologies that provide different models. They've been coming for years: Python, Ruby, Java, Perl and others have been putting together their own commons of software. They have all been working to provide virtualization to isolate one program (and its dependencies) from another. Containerization takes that to the next level. Sometimes that's what our users want.
We haven't figured out what the balances are, how we fit into this new world. However, I disagree with the claim that we aren't even discussing the problem. I think we're trying a lot of things off in our own little technical groups. We're just getting to the level of critical mass of understanding where we can take advantage of Debian's modern form of innovation.
Because here's the thing. Debian's innovation now is social, not technical. Just as we're in the middle technically, we're in the middle socially. Upstreams, developers, users, derivatives, and all the other members of our community work together. we're a place where we can share technology, explore solutions, and pull apart common elements. This is the first Debconf where it felt like we'd explored some of these trends enough to start understanding how they might fit together in a whole. Seven years ago, it felt like we were busy being convinced the Java folks were wrong-headed. A couple of years later, it felt like we were starting to understand our users' desires that let to models different than packaging, but we didn't have any thoughts. At least in my part of the hallway it sounded like people were starting to think about how they might fit parts together and what the tradeoffs would be.
Yes, Matthew's talk doubtless sparked some of that. I think he gave us a well deserved and important wake-up call. However, I was excited by the discussion prior to Thursday.
What I'm taking a way is that Debian is valuable when there's a role in the middle. Both socially and technically we should capitalize on situations where something between makes things better and get out of the way where it does not.

9 April 2017

Sam Hartman: When "when" is too hard a question: SQLAlchemy, Python datetime, and ISO8601

A new programmer asked on a work chat room how timezones are handled in databases. He asked if it was a good idea to store things in UTC. The senior programmers all laughed as we told some of our horror stories with timezones. Yes, UTC is great; if only it were that simple.
About a week later I was designing the schema for a blue sky project I'm implementing. I had to confront time in all its Pythonic horror.
Let's start with the datetime.datetime class. Datetime objects optionally include a timezone. If no timezone is present, several methods such as timestamp treat the object as a local time in the system's timezone. The timezone method returns a POSIX timestamp, which is always expressed in UTC, so knowing the input timezone is important. The now method constructs such an object from the current time.
However other methods act differently. The utcnow method constructs a datetime object that has the UTC time, but is not marked with a timezone. So, for example datetime.fromtimestamp(datetime.utcnow().timestamp()) produces the wrong result unless your system timezone happens to have the same offset as UTC.
It's also possible to construct a datetime object that includes a UTC time and is marked as having a UTC time. The utcnow method never does this, but you can pass the UTC timezone into the now method and get that effect. As you'd expect, the timestamp method returns the correct result on such a datetime.
Now enter SQLAlchemy, one of the more popular Python ORMs. Its DATETIME type has an argument that tries to request a column capable of storing a a timezone from the underlying database. You aren't guaranteed to get this though; some databases don't provide that functionality. With PostgreSQL, I do get such a column, although something in SQLAlchemy is not preserving the timezones (although it is correctly adjusting the time). That is, I'll store a UTC time in an object, flush it to my session, and then read back the same time represented in my local timezone (marked as my local timezone). You'd think this would be safe.
Enter SQLite. SQLite makes life hard for people wanting to store time; it seems to want to store things as strings. That's fairly incompatible with storing a timezone and doing any sort of comparisons on dates. SQLAlchemy does not try to store a timezone in SQLite. It just trims any timezone information from the datetime. So, if I do something like
d = datetime.now(timezone.utc)
obj.date_col = d
session.add(obj)
session.flush()
assert obj.date_col == d # fails
assert obj.date_col.timestamp() == d.timestamp() # fails
assert d == obj.date_col.replace(tzinfo = timezone.utc) # finally succeeds

There are some unfortunate consequences of this. If you mark your datetimes with timezone information (even if it is always the same timezone), whether two datetimes representing the same datetime compare equal depends on whether objects have been flushed to the session yet. If you don't mark your objects with timezones, then you may not store timezone information on other databases.
At least if you use only the methods we've discussed so far, you're reasonably safe if you use local time everywhere in your application and don't mark your datetimes with timezones. That's undesirable because as our new programmer correctly surmised, you really should be using UTC. This is particularly true if users of your database might span multiple timezones.
You can use UTC time and not mark your objects as UTC. This will give the wrong data with a database that actually does support timezones, but will sort of work with SQLite. You need to be careful never to convert your datetime objects into POSIX time as you'll get the wrong result.
It turns out that my life was even more complicated because parts of my project serialize data into JSON. For that serialization, I've chosen ISO 8601. You've probably seen that format: '2017-04-09T18:17:27.340410+00:00. Datetime provides the convenient isoformat method to print timestamps in the ISO 8601 format. If the datetime has a timezone indication, it is included in the ISO formatted string. If not, then no timezone indication is included. You know how I mentioned that datetime takes a string without a timezone marker as local time? Yeah, well, that's not what 8601 does: UTC all the way, baby! And at least the parser in the iso8601 module will always include timezone markers. So, if you use datetime to print a timestamp without a timezone marker and then read that back in to construct a new datetime on the deserialization side, then you'll get the wrong time. OK, so mark things with timezones then. Well, if you use local time, then the time you get depends on whether you print the ISO string before or after session flush (before or after SQLAlchemy trims the timezone information as it goes to SQLite).
It turns out that I had the additional complication of one side of my application using SQLite and one side using PostgreSQL. Remember how I mentioned that something between SQLAlchemy and PostgreSQL was recasting my times in local timezone (although keeping the time the same)? Well, consider how that's going to work. I serialize with the timezone marker on the PostgreSQL side. I get a ISO8601 localtime marked with the correct timezone marker. I deserialize on the SQLite side. Before session flush, I get a local time marked as localtime. After session flush, I get a local time with no marking. That's bad. If I further serialize on the SQLite side, I'll get that local time incorrectly marked as UTC. Moreover, all the times being locally generated on the SQLite side are UTC, and as we explored, SQLite really only wants one timezone in play.
I eventually came up with the following approach:

  1. If I find myself manipulating a time without a timezone marking, assert that its timezone is UTC not localtime.

  2. Always use UTC for times coming into the system.

  3. If I'm generating an ISO 8601 time from a datetime that has a timezone marker in a timezone other than UTC, represent that time as a UTC-marked datetime adjusting the time for the change in timezone.


This is way too complicated. I think that both datetime and SQLAlchemy's SQLite time handling have a lot to answer for. I think SQLAlchemy's core time handling may also have some to answer for, but I'm less sure of that.

29 January 2017

Sam Hartman: Network Audio Visualization: Network Modeling

Previously, I wrote about my project to create an audio depiction of network traffic. In this second post, I explore how I model aspects of the network that will be captured in the audio representation. Before getting started, I'll pass along a link. This is not the first time someone has tried to put sound to packets flying through the ether: I was pointed at Peep. I haven't looked at Peep, but will do so after I finish my own write up. Not being an academic, I feel no obligation to compare and contrast my work to others:-)
I started with an idea of what I'd like to hear. One of my motivations was to explore some automated updates we run at work. So, I was hoping to capture the initial DNS and ARP traffic as the update discovered the systems it would contact. Then I was hoping to capture the ssh and other traffic of the actual update.
To Packet or Stream
One of the simplest things to do would simply be to model network packets. For DNS I chose that approach. I was dubious that a packet-based model would capture the aspects of TCP streams I typically care about. I care about the source and destination (both address and port) of course. However I also care about how much traffic is being carried over the stream and the condition of the stream. Are there retransmits? Are there a bunch of unanswered SYNs? But I don't care about the actual distribution of packets. Also, a busy TCP stream can generate thousands of packets a second. I doubted my ability to distinguish thousands of sounds a second at all, especially while trying to convey enough information to carry stream characteristics like overall traffic volume.
So, for TCP, I decided to model some characteristics of streams rather than individual packets.
For DNS, I decided to represent individual requests/replies.
I came up with something clever for ARPP. There, I model the request/reply as an outstanding request. A lot of unanswered ARPs can be a sign of a scan or a significant problem. The mornful sound of a TCP stream trailing off into an unanswered ARP as the cache times out on a broken network is certainly something I'd like to capture. So, I track when an ARP request is sent and when/if it is answered.
Sound or Music
I saw two approaches. First, I could use some sound to represent streams. As an example, a running diesel engine could make a great representation of a stream. The engine speed could represent overall traffic flow. There are many opportunities for detuning the engine to represent various problems that can happen with a stream. Perhaps using stereo separation and slightly different fundamental frequencies I could even represent a couple of streams and still be able to track them.
However, at least with me as a listener, that's not going to scale to a busy network. The other option I saw was to try and create melodic music with various musical phrases modified as conditions within the stream or network changed. That seemed a lot harder to do, but humans are good at listening to complicated music.
I ended up deciding that at least for the TCP streams, I was going to try and produce something more musical than sound. I was nervous: I kept having visions of a performance of "Peter and the Wolf" with different instruments representing all the characters that somehow went dreadfully wrong.
As an aside, the decision to approach music rather than sound depended heavily on what I was trying to capture. If I'm modeling more holistic properties of a system--for example, total network traffic without splitting into streams--I think parameterized sounds would be a better approach.
The decision to approach things musically affected the rest of the modeling. Somehow I was going to need to figure out notes to play. I'd already rejected the idea of modeling packets, so I wouldn't simply be able to play notes when a packet arrived.
Energy Decay
As I played with various options, I realized that the critical challenge would be figuring out how to focus the listener's attention on the important aspects of what was going on. Clutter was the great enemy. My job would be figuring out how to spend sound wisely. When something interesting happened, that part of the model should get more focus--more of the listener's energy.
Soon I found myself thinking a lot about managing the energy of network streams. I imagined streams getting energy when something happened, and spending that energy to convey that interesting event to the listener. Energy needed to accumulate fast enough that even low-traffic streams could be noticed. Energy needed to be spent fast enough that old events were not taking listener focus from new, interesting things going on. However, if the energy were spent slow enough, then network events could be smoothed out to give a better picture of the stream rather than individual packets.
This concept of managing some decaying quantity and managing the rate of decay proved useful at multiple levels of the model.
Two Layer Model
I started with a python script that parses tcpdump output. It associates a packet with a stream and batches packets together to avoid overloading other parts of the system.
The output of this script are stream events. Events include a source and destination address, a stream ID, traffic in each direction, and any special events on the stream.
For DNS, the script just outputs packet events. For ARP, the script outputs request start, reply, and timeout events. There's some initial support for UDP, but so far that doesn't make sound.
Right now, FINs are modeled, but SYNs and the interesting TCP conditions aren't directly modeled. If you get retransmissions you'll notice because packet flow will decrease. However, I'd love to explicitly sound retransmissions. I also think a window filling as an application fails to read is important. I imagine either narrowing a band-pass filter to clamp the audio bandwidth available to a stream with a full window. Or perhaps taking it the other direction and adding an echo.
The next layer down tracks the energy of each stream. But that, and how I map energy into music, is the topic of the next post.

23 January 2017

Sam Hartman: Cudos to GDB Contributors

I recently diagnosed a problem in Debian's pam-p11 package. This package allegedly permits logging into a computer using a smart card or USB security token containing an ssh key. If you know the PIN and have the token, then your login attempt is authorized against the ssh authorized keys file. This seems like a great way to permit console logins as root to machines without having a shared password. Unfortunately, the package didn't work very well for me. It worked once, then all future attempts to use it segfaulted. I'm familiar with how PAM works. I understand the basic ideas behind PKCS11 (the API used for this type of smart card), but was completely unfamiliar with this particular PAM module and the PKCS11 library it used. The segfault was in an area of code I didn't even expect that this PAM module would ever call. Back in 1994, that would have been a painful slog. Gdb has improved significantly since then, and I'd really like to thank all the people over the years who made that possible. I was able to isolate the problem in just a couple of hours of debugging. Here are some of the cool features I used: Anyway, with just a couple of hours and no instrumentation of the code, I managed to track down how a bunch of structures were being freed as an unexpected side effect of one of the function calls. Neither I nor the author of the pam-p11 module expected that (although it is documented and does make sense in retrospect). Good tools make life easier.

15 January 2017

Sam Hartman: Musical Visualization of Network Traffic

I've been working on a fun holiday project in my spare time lately. It all started innocently enough. The office construction was nearing its end, and it was time for my workspace to be set up. Our deployment wizard and I were discussing. Normally we stick two high-end monitors on a desk. I'm blind; that seemed silly. He wanted to do something similarly nice for me, so he replaced one of the monitors with excellent speakers. They are a joy to listen to, but I felt like I should actually do something with them. So, I wanted to play around with some sort of audio project.
I decided to take a crack at an audio representation of network traffic. The solaris version of ping used to have an audio option, which would produce sound for successful pings. In the past I've used audio queues to monitor events like service health and build status.
It seemed like you could produce audio to give an overall feel for what was happening on the network. I was imagining a quick listen would be able to answer questions like:

  1. How busy is the network

  2. How many sources are active

  3. Is the traffic a lot of streams or just a few?

  4. Are there any interesting events such as packet loss or congestion collapse going on?

  5. What's the mix of services involved


I divided the project into three segments, which I will write about in future entries:

I'm fairly happy with what I have. It doesn't represent all the items above. As an example, it doesn't directly track packet loss or retransmissions, nor does it directly distinguish one service from another. Still, just because of the traffic flow, rsync sounds different from http. It models enough of what I'm looking for that I find it to be a useful tool. And I learned a lot about music and Linux audio. I also got to practice designing discrete-time control functions in ways that brought back the halls of MIT.

1 January 2017

Sam Hartman: 2016

I was in such a different place at the beginning of 2016: I was poised to continue to work to help the world find love. Professionally, I was ready to make a much needed transition and find new projects to work on. The year 2016 sucked. It feels like the year was filled with many different versions of the universe saying "Not interested in what you have to offer." At the beginning of the year, I had the energy to try and reach across large disagreements and help find common ground even when compromise was not possible. Now, my blog lies fallow because I cannot find the strength to be vulnerable enough to write what I would choose to say. Certainly a lot of the global changes of the last year have felt like a strong rejection of the world I'd like to see. However, many of the rejections have been personal. Beyond that, most of the people who stood as pillars of support in my life, together helping me find the strength to be vulnerable, are no longer available. When the universe sends such strong messages, it's a good idea to ask whether you are on the right path. I certainly have discovered training I need and things I need to improve in order to avoid making costly mistakes that hurt others. However, among the rejections were clear demonstrations of the value of reaching out with love and compassion. Besides, this is what I'm called to do. It's what I want to do. I certainly will not force it on anyone. But it looks like the next few years may be a hard struggle to find pockets of people interested in that work, finding people who will choose love even in the current world, along with some difficult training to learn from challenges of the last year. Amongst all this, my life if filled with love. There are new connections even as old connections are strained. There is always the hope of finding new ways to connect when the old ones are no longer right. I will rebuild and regain safety. I have the tools to do that. The process is just long and complicated.

6 October 2006

Andree Leidenfrost: Easy Peasy AFS on Debian

Bug #385790 prompted me to set up an AFS client on my sid installation. This is something I hadn't done since I left uni almost nine years ago. Back then it was a bit fiddly to get the Transarc AFS client for Linux to work if I remember correctly.

Things have quite obviously improved since then. The following is all that was required (with some helpful information provided by the submitter - thanks, Kevin!):
  • install packages openafs-modules-source and module-assistant
  • follow the instructions in /usr/share/doc/openafs-modules-source/README.modules, i.e:
    • module-assistant prepare openafs-modules
    • module-assistant auto-build openafs-modules
    • dpkg -i /usr/src/openafs-modules-.deb
  • install package openafs-client and configure like this:
    • leave AFS cell of workstation at default, i.e. local domain
    • leave cache at 50000 kb
    • leave DB server host for home cell blank
    • answer 'Yes' to 'Run Openafs client now and at boot?'
  • if this doesn't work, run dpkg-reconfigure openafs-client like this:
    • leave AFS cell of workstation at default, i.e. local domain
    • leave cache at 50000 kb
    • answer 'Yes' to 'Run Openafs client now and at boot?'
    • answer 'Yes' to 'Look up AFS cells in DNS?'
    • answer 'Yes' to 'Encrypt authenticated traffic with AFS fileserver?'
    • answer 'Yes' to 'Dynamically generate the contents of /afs?'
    • answer 'Yes' to 'Use fakestat to avoid hangs when listing /afs?'
    • leave DB server host for home cell blank
    • answer 'Yes' to 'Run Openafs client now and at boot?' (again)
  • ...and restart openafs-client afterwards
(The need to reconfigure may be a bug, not sure.)

In summary, OpenAFS and Sam Hartman's packaging effort make AFS a breeze to install on Debian!

Now all I have to do is find the time to fix the bug. ;-)

[Update] Important detail I forgot to mention: Open port 7001 on your firewall for UDP.
[Update] Added what to answer, i.e. 'Yes'. Doh.

19 March 2006

Clint Adams: This report is flawed, but it sure is fun

91D63469DFdnusinow1243
63DEB0EC31eloy
55A965818Fvela1243
4658510B5Amyon2143
399B7C328Dluk31-2
391880283Canibal2134
370FE53DD9opal4213
322B0920C0lool1342
29788A3F4Cjoeyh
270F932C9Cdoko
258768B1D2sjoerd
23F1BCDB73aurel3213-2
19E02FEF11jordens1243
18AB963370schizo1243
186E74A7D1jdassen(Ks)1243
1868FD549Ftbm3142
186783ED5Efpeters1--2
1791B0D3B7edd-213
16E07F1CF9rousseau321-
16248AEB73rene1243
158E635A5Erafl
14C0143D2Dbubulle4123
13D87C6781krooger(P)4213
13A436AD25jfs(P)
133D08B612msp
131E880A84fjp4213
130F7A8D01nobse
12F1968D1Bdecklin1234
12E7075A54mhatta
12D75F8533joss1342
12BF24424Csrivasta1342
12B8C1FA69sto
127F961564kobold
122A30D729pere4213
1216D970C6eric12--
115E0577F2mpitt
11307D56EDnoel3241
112BE16D01moray1342
10BC7D020Aformorer-1--
10A7D91602apollock4213
10A51A4FDDgcs
10917A225Ejordi
104B729625pvaneynd3123
10497A176Dloic
962F1A57Fpa3aba
954FD2A58glandium1342
94A5D72FErafael
913FEFC40fenio-1--
90AFC7476rra1243
890267086duck31-2
886A118E6ch321-
8801EA932joey1243
87F4E0E11waldi-123
8514B3E7Cflorian21--
841954920fs12--
82A385C57mckinstry21-3
825BFB848rleigh1243
7BC70A6FFpape1---
7B70E403Bari1243
78E2D213Ajochen(Ks)
785FEC17Fkilian
784FB46D6lwall1342
7800969EFsmimram-1--
779CC6586haas
75BFA90ECkohda
752B7487Esesse2341
729499F61sho1342
71E161AFBbarbier12--
6FC05DA69wildfire(P)
6EEB6B4C2avdyk-12-
6EDF008C5blade1243
6E25F2102mejo1342
6D1C41882adeodato(Ks)3142
6D0B433DFross12-3
6B0EBC777piman1233
69D309C3Brobert4213
6882A6C4Bkov
66BBA3C84zugschlus4213
65662C734mvo
6554FB4C6petere-1-2
637155778stratus
62D9ACC8Elars1243
62809E61Ajosem
62252FA1Afrank2143
61CF2D62Amicah
610FA4CD1cjwatson2143
5EE6DC66Ajaldhar2143
5EA59038Esgran4123
5E1EE3FB1md4312
5E0B8B2DEjaybonci
5C9A5B54Esesse(Ps,Gs) 2341
5C4CF8EC3twerner
5C2FEE5CDacid213-
5C09FD35Atille
5C03C56DFrfrancoise---1
5B7CDA2DCxam213-
5A20EBC50cavok4214
5808D0FD0don1342
5797EBFABenrico1243
55230514Asjackman
549A5F855otavio-123
53DC29B41pdm
529982E5Avorlon1243
52763483Bmkoch213-
521DB31C5smr2143
51BF8DE0Fstigge312-
512CADFA5csmall3214
50A0AC927lamont
4F2CF01A8bdale
4F095E5E4mnencia
4E9F2C747frankie
4E9ABFCD2devin2143
4E81E55C1dancer2143
4E38E7ACFhmh(Gs)1243
4E298966Djrv(P)
4DF5CE2B4huggie12-3
4DD982A75speedblue
4C671257Ddamog-1-2
4C4A3823Ekmr4213
4C0B10A5Bdexter
4C02440B8js1342
4BE9F70EAtb1342
4B7D2F063varenet-213
4A3F9E30Eschultmc1243
4A3D7B9BClawrencc2143
4A1EE761Cmadcoder21--
49DE1EEB1he3142
49D928C9Bguillem1---
49B726B71racke
490788E11jsogo2143
4864826C3gotom4321
47244970Bkroeckx2143
45B48FFAEmarga2143
454E672DEisaac1243
44B3A135Cerich1243
44597A593agmartin4213
43FCC2A90amaya1243
43F3E6426agx-1-2
43EF23CD6sanvila1342
432C9C8BDwerner(K)
4204DDF1Baquette
400D8CD16tolimar12--
3FEC23FB2bap34-1
3F972BE03tmancill4213
3F801A743nduboc1---
3EBEDB32Bchrsmrtn4123
3EA291785taggart2314
3E4D47EC1tv(P)
3E19F188Etroyh1244
3DF6807BEsrk4213
3D2A913A1psg(P)
3D097A261chrisb
3C6CEA0C9adconrad1243
3C20DF273ondrej
3B5444815ballombe1342
3B1DF9A57cate2143
3AFA44BDDweasel(Ps,Gs) 1342
3AA6541EEbrlink1442
3A824B93Fasac3144
3A71C1E00turbo
3A2D7D292seb128
39ED101BFmbanck3132
3969457F0joostvb2143
389BF7E2Bkobras1--2
386946D69mooch12-3
374886B63nathans
36F222F1Fedelhard
36D67F790foka
360B6B958geiger
3607559E6mako
35C33C1B8dirson
35921B5D8ajmitch
34C1A5BE5sjq
3431B38BApxt312-
33E7B4B73lmamane2143
327572C47ucko1342
320021490schepler1342
31DEB8EAEgoedson
31BF2305Akrala(Gs)3142
319A42D19dannf21-4
3174FEE35wookey3124
3124B26F3mfurr21-3
30A327652tschmidt312-
3090DD8D5ingo3123
30813569Fjeroen1141
30644FAB7bas1332
30123F2F2gareuselesinge1243
300530C24bam1234
2FD6645ABrmurray-1-2
2F95C2F6Dchrism(P)
2F9138496graham(Gs)3142
2F5D65169jblache1332
2F28CD102absurd
2F2597E04samu
2F0B27113patrick
2EFA6B9D5hamish(P)3142
2EE0A35C7risko4213
2E91CD250daigo
2D688E0A7qjb-21-
2D4BE1450prudhomm
2D2A6B810joussen
2CFD42F26dilinger
2CEE44978dburrows1243
2CD4C0D9Dskx4213
2BFB880A3zeevon
2BD8B050Droland3214
2B74952A9alee
2B4D6DE13paul
2B345BDD3neilm1243
2B28C5995bod4213
2B0FA4F49schoepf
2B0DDAF42awoodland
2A8061F32osamu4213
2A21AD4F9tviehmann1342
299E81DA0kaplan
2964199E2fabbe3142
28DBFEC2Fpelle
28B8D7663ametzler1342
28B143975martignlo
288C7C1F793sam2134
283E5110Fovek
2817A996Atfheen
2807CAC25abi4123
2798DD95Cpiefel
278D621B4uwe-1--
26FF0ABF2rcw2143
26E8169D2hertzog3124
26C0084FCchrisvdb
26B79D401filippo-1--
267756F5Dfrn2341
25E2EB5B4nveber123-
25C6153ADbroonie1243
25B713DF0djpig1243
250ECFB98ccontavalli(Gs)
250064181paulvt
24F71955Adajobe21-3
24E2ECA5Ajmm4213
2496A1827srittau
23E8DCCC0maxx1342
23D97C149mstone(P)2143
22DB65596dz321-
229F19BD1meskes
21F41B907marillat1---
21EB2DE66boll
21557BC10kraai1342
2144843F5lolando1243
210656584voc
20D7CA701steinm
205410E97horms
1FC992520tpo-14-
1FB0DFE9Bgildor
1FAEEB4A9neil1342
1F7E8BC63cedric21--
1F2C423BCzack1332
1F0199162kreckel4214
1ECA94FA8ishikawa2143
1EAAC62DFcyb---1
1EA2D2C41malattia-312
1E77AC835bcwhite(P)
1E66C9BB0tach
1E145F334mquinson2143
1E0BA04C1treinen321-
1DFE80FB2tali
1DE054F69azekulic(P)
1DC814B09jfs
1CB467E27kalfa
1C9132DDByoush-21-
1C87FFC2Fstevenk-1--
1C2CE8099knok321-
1BED37FD2henning(Ks)1342
1BA0A7EB5treacy(P)
1B7D86E0Fcmb4213
1B62849B3smarenka2143
1B3C281F4alain2143
1B25A5CF1omote
1ABA0E8B2sasa
1AB474598baruch2143
1AB2A91F5troup1--2
1A827CEDEafayolle(Gs)
1A6C805B9zorglub2134
1A674A359maehara
1A57D8BF7drew2143
1A269D927sharky
1A1696D2Blfousse1232
19BF42B07zinoviev--12
19057B5D3vanicat2143
18E950E00mechanix
18BB527AFgwolf1132
18A1D9A1Fjgoerzen
18807529Bultrotter2134
1872EB4E5rcardenes
185EE3E0Eangdraug12-3
1835EB2FFbossekr
180C83E8Eigloo1243
17B8357E5andreas212-
17B80220Dsjr(Gs)1342
17796A60Bsfllaw1342
175CB1AD2toni1---
1746C51F4klindsay
172D03CB1kmuto4231
171473F66ttroxell13-4
16E76D81Dseanius1243
16C63746Dhector
16C5F196Bmalex4213
16A9F3C38rkrishnan
168021CE4ron---1
166F24521pyro-123
1631B4819anfra
162EEAD8Bfalk1342
161326D40jamessan13-4
1609CD2C0berin--1-
15D8CDA7Bguus1243
15D8C12EArganesan
15D64F870zobel
159EF5DBCbs
157F045DCcamm
1564EE4B6hazelsct
15623FC45moronito4213
1551BE447torsten
154AD21B5warmenhoven
153BBA490sjg
1532005DAseamus
150973B91pjb2143
14F83C751kmccarty12-3
14DB97694khkim
14CD6E3D2wjl4213
14A8854E6weinholt1243
14950EAA6ajkessel
14298C761robertc(Ks)
142955682kamop
13FD29468bengen-213
13FD25C84roktas3142
13B047084madhack
139CCF0C7tagoh3142
139A8CCE2eugen31-2
138015E7Ethb1234
136B861C1bab2143
133FC40A4mennucc13214
12C0FCD1Awdg4312
12B05B73Arjs
1258D8781grisu31-2
1206C5AFDchewie-1-1
1200D1596joy2143
11C74E0B7alfs
119D03486francois4123
118EA3457rvr
1176015EDevo
116BD77C6alfie
112AA1DB8jh
1128287E8daf
109FC015Cgodisch
106468DEBfog--12
105792F34rla-21-
1028AF63Cforcer3142
1004DA6B4bg66
0.zufus-1--
0.zoso-123
0.ykomatsu-123
0.xtifr1243
0.xavier-312
0.wouter2143
0.will-132
0.warp1342
0.voss1342
0.vlm2314
0.vleeuwen4312
0.vince2134
0.ukai4123
0.tytso-12-
0.tjrc14213
0.tats-1-2
0.tao1--2
0.stone2134
0.stevegr1243
0.smig-1-2
0.siggi1-44
0.shaul4213
0.sharpone1243
0.sfrost1342
0.seb-21-
0.salve4213
0.ruoso1243
0.rover--12
0.rmayr-213
0.riku4123
0.rdonald12-3
0.radu-1--
0.pzn112-
0.pronovic1243
0.profeta321-
0.portnoy12-3
0.porridge1342
0.pmhahn4123
0.pmachard1--2
0.pkern3124
0.pik1--2
0.phil4213
0.pfrauenf4213
0.pfaffben2143
0.p21243
0.ossk1243
0.oohara1234
0.ohura-213
0.nwp1342
0.noshiro4312
0.noodles2134
0.nomeata2143
0.noahm3124
0.nils3132
0.nico-213
0.ms3124
0.mpalmer2143
0.moth3241
0.mlang2134
0.mjr1342
0.mjg591342
0.merker2--1
0.mbuck2143
0.mbrubeck1243
0.madduck4123
0.mace-1-2
0.luther1243
0.luigi4213
0.lss-112
0.lightsey1--2
0.ley-1-2
0.ldrolez--1-
0.lange4124
0.kirk1342
0.killer1243
0.kelbert-214
0.juanma2134
0.jtarrio1342
0.jonas4312
0.joerg1342
0.jmintha-21-
0.jimmy1243
0.jerome21--
0.jaqque1342
0.jaq4123
0.jamuraa4123
0.iwj1243
0.ivan2341
0.hsteoh3142
0.hilliard4123
0.helen1243
0.hecker3142
0.hartmans1342
0.guterm312-
0.gniibe4213
0.glaweh4213
0.gemorin4213
0.gaudenz3142
0.fw2134
0.fmw12-3
0.evan1--2
0.ender4213
0.elonen4123
0.eevans13-4
0.ean-1--
0.dwhedon4213
0.duncf2133
0.ds1342
0.dparsons1342
0.dlehn1243
0.dfrey-123
0.deek1--2
0.davidw4132
0.davidc1342
0.dave4113
0.daenzer1243
0.cupis1---
0.cts-213
0.cph4312
0.cmc2143
0.clebars2143
0.chaton-21-
0.cgb-12-
0.calvin-1-2
0.branden1342
0.brad4213
0.bnelson1342
0.blarson1342
0.benj3132
0.bayle-213
0.baran1342
0.az2134
0.awm3124
0.atterer4132
0.andressh1---
0.amu1--2
0.akumria-312
0.ajt1144
0.ajk1342
0.agi2143
0.adric2143
0.adejong1243
0.adamm12--
0.aba1143