Search Results: "mattb"

12 July 2023

Matt Brown: 2023 Mid Year Review

I m six months into my journey of building a business which means its time to reflect and review the goals I set for the year.

No further investment in co2mon.nz In March I made the decision to focus on completing the market research for co2mon.nz. The results of that research led to two key conclusions:
  1. Indoor air quality/ventilation is not a problem many people are actively thinking about or looking to spend money to improve.
  2. Even when introduced to the problem and educated about the need, most people are looking for a one-off expense or solution (e.g. the physical monitor) and are much less interested in a monitoring software service.
Based on that, it was clear that this is not an opportunity that I should continue pursuing and I ve put co2mon.nz into maintenance mode. I ve committed to maintaining the infrastructure to support existing customers, but I won t be investing time or energy in developing it further.

Discipline in selecting product opportunities The decision to stop investing more time into co2mon.nz was straightforward given the results of the research, but it was also painful given the time I ve already sunk into it. I hindsight it s clear that my enthusiasm to solve a problem with technology I enjoyed was my driving force rather than a deep understanding of the wants and needs of potential customers. I don t entirely regret trying my luck once - but it s not time efficient and I know that following that pattern again is not a sustainable or viable path to building a successful business. I ve decided to use the following list of questions to bring more discipline to how I evaluate product opportunities in future:
  1. Problem: Is this something that a sizeable number of people are struggling with AND are willing to spend money solving?
  2. Capability: Can I deliver a solution that solves the problem in a reliable and cost-effective way?
  3. Excitement: Am I excited and motivated to invest time in building the solution to this problem?
  4. Trust: Do I have the expertise and experience to be trusted to solve the problem by potential customers?
  5. Execution: Can I package, market and sell that solution in a profitable manner?
My plan is to answer these questions and then make an evaluation of the potential before I commit time to building any part of a product. As an example of how I think that will help, here s what I think the answer to those questions for ventilation monitoring would have been:
  1. Problem: No - as the market research eventually showed.
  2. Capability: Low - The part of the solution which customers primarily value (the hardware) is complex and outside of my core experience. The software I can easily deliver is not where the value is seen.
  3. Excitement: Yes - this was the primary driver of starting the project.
  4. Trust: Low - I m trusted to build software, but cannot claim any specific expertise in air quality and ventilation.
  5. Execution: Low confidence - These skills are not ones I ve exercised a lot in my career to date.
What these answers point to is that identifying the problem alone is not enough. I don t expect every question to have a perfect answer, but I want to hold myself to only pursuing opportunities where there s only one major area of doubt. In this case, even had the market research demonstrated a problem that many customers would pay to solve, there were still some big answers missing to the trust, capability and execution questions. Overall my conclusion is that co2mon.nz was not the ideal business to start my journey with given the number of open questions in the plan. I like to think that conclusion would also have been clear to me six months ago had I taken the time to go through this process then!

Prioritising areas of growth Applying those questions to my other product ideas results in a lot of I don t know yet answers to the problem and capability questions, further reinforcing the lesson that I need to spend more understanding if there is a problem with a viable business model attached in those areas before progressing any of those ideas. Beyond that lesson, a more interesting observation comes from the last question regarding execution. My answers to the first four questions vary between ideas, but my answers to this last question are always the same - I don t have a lot of confidence in my sales and marketing skills to sell a product. That s not a surprise. My career to date has been focused on software development and leadership, I don t have a lot of experience with sales and marketing. The opportunity to grow and develop those skills is actually a large part of my motivation for choosing the path of building my own business. But seeing that this is a common factor that will need significant investment regardless of which opportunity I pursue sends me a strong signal that I should focus on growth in this area as a priority. Following that logic through to the next step of what creating that focus would look like reveals a conflict: The nature of the mission I ve set for myself draws me to products in areas that are new to me, which means there s also a need to invest in building expertise in those areas. Again not a surprise, but the time and focus required to develop that expertise competes with time spent growing my sales and marketing skills. So I have a prioritisation problem. Solving it is going to require changing the type of product I m trying to build in the short term: I need to build a product that uses my existing expertise and strengths as much as possible, so that I can put the majority of my energy into growing the core business skills where my confidence is currently lacking. Trying to deliver meaningful improvements to a big problem in an area I don t have past experience in while also learning how to sell and market a product is biting off more than I chew right away.

Changing the goal posts With those lessons in hand I m making three changes to my 2023 Goals:
  1. Reducing the product development goal from several ideas to two. The first was co2mon.nz. The second will be drawn from my existing expertise - not one of the previously stated ideas that require me to develop expertise in a new area.
  2. Moving the consulting and product development goals to be alternatives - I expect I can achieve at most one of them this year.
  3. Reducing the publishing target for this site from at least once a week to once a month . I thought I d have more to say this year, but the words are coming very slowly to me.
Reducing scope and ambition is humbling, but that s reality. I hope it turns out to be a case of slow down and lay the foundations in order to then move faster. The good news is that I don t feel the need to make any changes to the vision, mission and strategy I m following - I think they re still the right destination and overall path for me even though the first six months has proven bumpy. I just need to be a bit more realistic on the short-term goals that will feed into them.

The next few months I m choosing to prioritise the product development goal. I m aiming to complete the market research/problem definition phase for a product opportunity I ve identified in the SRE/DevOps space (where my previous experience is) and make a decision on whether to start development by mid August. In making that decision I plan to gather the answers to my questions, and then diligently evaluate whether the opportunity is worth committing to or not. I will write more about this process in coming weeks. If I decide to proceed that gives me 2-3 months to get an MVP in the hands of customers and get concrete validation of whether the product has revenue and growth potential before the end of the year. Tight, but if things go well, and I don t take any further consulting work, there s a reasonable chance I can complete the revised goal successfully. In the event that I decide the product opportunity I m currently researching is not the right one to commit to, I will likely revert to focusing on my consulting goal in the remaining 2-3 months of the year rather than attempt a third product development iteration. Thanks for reading this far! As always, I d love your thoughts and feedback.

Appendix: Revised 2023 Goals Putting all that together, the ultimate outcome of this review (including updated progress scoring) looks like:
  1. Execute a series of successful consulting engagements, building a reputation for myself and leaving happy customers willing to provide testimonials that support a pipeline of future opportunities. Score: 3/10 - I focused entirely on co2mon.nz during April, May and June to the detriment of my pipeline of consulting work. This score is unlikely to improve given the above plan unless I decide not to commit to developing the idea I m currently investigating.
  2. Grow my product development skill set by taking two ideas (co2mon.nz, an SRE/DevOps focused product) to MVP stage with customer feedback received, and generate revenue and has growth potential from one of them. Score: 4/10 - I launched co2mon.nz and got feedback, I discovered it didn t solve a problem relevant to customers and therefore did not generate substantial revenue or growth potential. Idea number two is in still in progress.
  3. Develop and maintain a broad professional network.
    1. To build a professional relationship with at least 30 new people this year. Score: 6/10 - This is going well. On track for a 10/10 score.
    2. To publish a piece of writing on this site once a month and for many of those to generate interesting conversations and feedback. Score: 6/10 - 4 out of 6 months have featured a post meeting this goal so far.
    3. To support the growth of my local technical community by volunteering my experience and knowledge with others. Score: 5/10 - I ve given one talk and helped with SREcon23 APAC, but not as much other work in this area as I d like.

27 June 2023

Matt Brown: Designing a PCBA friendly CO2 monitor

co2mon.nz currently uses monitors based on Oliver Seiler s open source design which I am personally building. This post describes my exploration of how to achieve production of a CO2 monitor that could enable the growth of co2mon.nz.

Goals Primarily I want to design a CO2 monitor which allows the majority of the production process to be outsourced. In particular, the PCB should be able to be assembled in an automated fashion (PCBA). As a secondary goal, I d like to improve the aesthetics of the monitor while retaining the unique feature of displaying clear visual indication of the current ventilation level through coloured lights. Overall, I ll consider the project successfull if I can achieve a visually attractive CO2 monitor which takes me less than 10 minutes per monitor to assemble/box/ship and whose production cost has the potential to be lower than the current model.

PCB

Schematic The existing CO2 monitor design provides a solid foundation but relies upon the ESP32 Devkit board, which is intended for evaluation purposes and is not well suited to automated assembly. Replacing this devkit board with the underlying ESP32 module is the major change needed to enable PCBA production, which then also requires moving the supporting electronics from the devkit board directly onto the primary PCB. The basic ESP32 chipset used in the devkit boards is no longer available as a discrete module suitable for placement directly onto a PCB which means the board will also have to be updated to use a more modern variant of the ESP32 chipset which is in active production such as the ESP32-S3. The ESP32-S3-WROOM1-N4 module is a very close match to the original devkit and will be suitable for this project. In addition to the change of ESP module, I made the following other changes to the components in use:
  • Added an additional temperature/humidity sensor (SHT30). The current monitors take temperature/humidity measurements from the SCD40 chipset. These are primarily intended to help in the calculation of CO2 levels and rely on an offset being subtracted to account for the heat generated by the electronic components themselves. I ve found their accuracy to OK, but not perfect. SHT30 is a cheap part, so its addition to hopefully provide improved temperature/humidity measurement is an easy choice.
  • Swapped to USB-C instead of USB-B for the power connector. USB-C is much more common than USB-B and is also smaller and not as tall off the board which provides more flexibility in the case design.
With major components selected the key task is to draw the schematic diagram describing how they electrically connect to each other, which includes all the supporting electronics (e.g. resistors, capacitors, etc) needed. Schematic I started out trying to use the EasyEDA/OSHWLab ecosystem thinking the tight integration with JLCPCB s assembly services would be a benefit, but the web interface was too clunky and limiting and I quickly got frustrated. KiCad proved to be a much more pleasant and capable tool for the job. The reference design in the ESP32 datasheet (p28) and USB-C power supply examples from blnlabs were particularly helpful alongside the KiCad documentation and the example of the existing monitor in completing this step (click the image to enlarge).

Layout The next step is to physically lay out where each component from the schematic will sit on the PCB itself. Obviously this requires first determining the overall size, shape and outline of the board and needs to occur in iteration with the intended design of the overall monitor, including the case, to ensure components like switches and USB sockets line up correctly. In addition to the requirements around the look and function of the case, the components themselves also have considerations that must be taken into account, including:
  • For best WiFi reception, the ESP32 antenna should be at the top of the monitor and should not have PCB underneath it, or for a specified distance either side of it.
  • The SHT30 temperature sensor should be as far from any heat generating components (e.g. the ESP32, BME680 and SCD40 modules) as possible and also considering that any generated heat will rise, as low on the monitor as possible.
  • The sensors measuring the air (SCD40, BME680 and SHT30) must have good exposure to the air outside the case.
PCB Taking all of these factors into account I ended up with a square PCB containing a cutout in the top right so that the ESP32 antenna can sit within the overall square outline while still meeting its design requirements. The SCD40 and BME680 sit in the top left corner, near the edges for good airflow and far away from the SHT30 temperature sensor in the bottom left corner. The LEDs I placed in a horizontal row across the center of the board, the LCD in the bottom right, a push button on the right-hand side and the USB-C socket in the center at the bottom. Once the components are placed, the next big task is to route the traces (aka wires) between the components on the board such that all the required electrical connections are made without any unintended connections (aka shorts) being created. This is a fun constraint solving/optimisation challenge and takes on an almost artistic aspect with other PCB designers often having strong opinions on which layout is best. The majority of the traces and routing for this board were able to be placed on the top layer of the PCB, but I also made use of the back layer for a few traces to help avoid conflicts and deal with places where different traces needed to cross each other. It s easy to see how this step would be much more challenging and time consuming on a larger and more complex PCB design. The final touches were to add some debugging breakouts for the serial and JTAG ports on the ESP32-S3 and a logo and various other helpful text on the silkscreen layer that will be printed on the PCB so it looks nice.

Production For assembly of the PCB, I went with JLCPCB based out of China. The trickiest part of the process was component selection and ensuring that the parts I had planned in the schematic were available. JLCPCB in conjunction with lcsc.com provides a basic and extended part library. If you use only basic parts you get quicker and cheaper assembly, while using extended parts bumps your order into a longer process with a small fee charged for each component on the board. Initially I spent a lot of time selecting components (particularly LEDs and switches) that were in the basic library before realising that the ESP32 modules are only available in the extended library! I think the lesson is that unless you re building the most trivial PCB with only passive components you will almost certainly end up in the advanced assembly process anyway, so trying to stay within the basic parts library is not worth the time. Unfortunately the SCD40 sensor, the most crucial part of the monitor, is not stocked at all by JLCPCB/LCSC! To work around this JLCPCB will maintain a personal component library for you when you ship components to them to for use in future orders. Given the extra logistical time and hassle of having to do this, combined with having a number of SCD40 components already on hand I decided to have the boards assembled without this component populated for the initial prototype run. This also had the benefit of lowering the risk if something went wrong as the cost of the SCD40 is greater than the cost of the PCB and all the other components combined! I found the kicad-jlcpcb-tools plugin for KiCad invaluable for keeping track of what part from lcsc.com I was planning to use for each component and generating the necessary output files for JLCPCB. The plugin allows you to store these mappings in your actual schematic which is very handy. The search interface it provides is fairly clunky and I found it was often easier to search for the part I needed on lcsc.com and then just copy the part number across into the plugin s search box rather than trying to search by name or component type. The LCD screen is the remaining component which is not easily assembled onto the PCB directly, but as you ll see next, this actually turned out to be OK as integrating the screen directly into the case makes the final assembly process smoother. fabricated PCBs The final surprise in the assembly process was the concept of edge rails, additional PCB material that is needed on either side of the board to help with feeding it through the assembly machine in the correct position. These can be added automatically by JLCPCB and have to be snapped off after the completed boards are received. I hadn t heard about these before and I was a little worried that they d interfere or get in the way of either the antenna cut-out at the top of the board, or the switch on the right hand side as it overhangs the edge so it can sit flush with the case. In the end there was no issue with the edge rails. The switch was placed hanging over them without issue and snapping them off once the boards arrived was a trivial 30s job using a vice to hold the edge rail and then gently tipping the board over until it snapped off - the interface between the board and the rails while solid looking has obviously been scored or perforated in some way during the production process so the edge breaks cleanly and smoothly. Magic! The process was amazingly quick with the completed PCBs (picture above) arriving within 7 days of the order being placed and looking amazing.

Case

Design I mocked up a very simple prototype of the case in FreeCAD during the PCB design process to help position and align the placement of the screen, switch and USB socket on the PCB as all three of these components interface directly with the edges of the case. Initially this design was similar to the current monitor design where the PCB (with lights and screen attached) sits in the bottom of the case, which has walls containing grilles for airflow and then a separate transparent perspex is screwed onto the top to complete the enclosure. As part of the aesthetic improvements for the new monitor I wanted to move away from a transparent front panel to something opaque but still translucent enough to allow the colour of the lights to show through. Without a transparent front panel the LCD also needs to be mounted directly into the case itself. The first few prototype iterations followed the design of the original CO2 monitor with a flat front panel that attaches to the rest of the case containing the PCB, but the new requirement to also attach the LCD to the front panel proved to make this unworkable. To stay in place the LCD has to be pushed onto mounting poles containing a catch mechanism which requires a moderate amount of force and applying that force to the LCD board when it is already connected to the PCB is essentially impossible. case with lcd attached As a result I ended up completely flipping the design such that the front panel is a single piece of plastic that also encompasses the walls of the case and contains appropriate mounting stakes for both the screen and the main PCB. Getting to this design hugely simplified the assembly process. Starting with an empty case lying face down on a bench, the LCD screen is pushed onto the mounting poles and sits flush with the cover of the case - easily achieved without the main PCB yet in place. case with pcb in place Next, the main PCB is gently lowered into the case facing downwards and sits on the mounting pole in each corner with the pins for the LCD just protruding through the appropriate holes in the PCB ready to be quickly soldered into place (this took significant iteration and tuning of dimensions/positioning to achieve!). Finally, a back panel can be attached which holds the PCB in place and uses cantilever snap joints to click on to the rest of the case. Overall the design is a huge improvement over the previous case which required screws and spacers to position the PCB and cover relative to the rest of the case, with the spacers and screws being particularly fiddly to work with. The major concern I had with the new design was that the mount to attach the monitor to the wall has moved from being attached to the main case and components directly to needing to be on the removable back panel - if the clips holding this panel to the case fail the core part of the monitor will fall off the wall which would not be good. To guard against this I ve doubled the size and number of clips at the top of the case (which bears the weight) and the result seems very robust in my testing. To completely assemble a monitor, including the soldering step takes me about 2-3 minutes individually, and would be even quicker if working in batches.

Production Given the number of design/testing iterations required to fine tune the case I chose not to outsource case production for now and used my 3D printer to produce them. I ve successfully used JLCPCB s 3D printing service for the previous case design, so I m confident that getting sufficient cases printed from JLCPCB or another supplier will not be an issue now that the design is finalised. completed monitor I tried a variety of filament colours, but settled on a transparent filament which once combined in the necessary layers to form the case is not actually transparent like perspex is, but provides a nice translucent medium which achieves the goal of having the light colour visible without exposing all of the circuit board detail. There s room for future improvement in the positioning of the LEDs on the circuit board to provide a more even distribution of light across the case but overall I really like the way the completed monitor ends up looking.

Evaluation Building this monitor has been a really fun project, both in seeing something progress from an idea, to plans on a screen to a nice physical thing on my wall, but also in learning and developing a bunch of new skills in PCB design, assembly and 3D design. completed monitor The goal of having a CO2 monitor which I can outsource the vast majority of production of is as close to being met as I think is possible without undertaking the final proof of placing a large order. I ve satisfied myself that each step is feasible and that the final assembly process is quick, easy and well below the level of effort and time it was taking me to produce the original monitors. Cost wise it s also a huge win, primarily in terms of the time taken, but also in the raw components - currently the five prototypes I ordered and built are on par with the component cost of the original CO2 monitor, but this will drop further with larger orders due to price breaks and amortisation of the setup and shipping expenses across more monitors. This project has also given me a much better appreciation for how much I m only just scratching the surface of the potential complexities and challenges in producing a hardware product of this type. I m reasonably confident I could successfully produce a few hundred and maybe even a few thousand monitors using this approach, but it s also clear that getting beyond that point is and would be a whole further level of effort and learning. Hardware is hard work. That s not news to anyone, including me, but there is something to be said for experiencing the process first hand to make the reality of what s required real. The PCB and case designs are both shared and can be found at https://github.com/co2monnz/co2monitor-pcb and https://github.com/co2monnz/cad, feedback and suggestions welcome!

13 June 2023

Matt Brown: Ventilation Monitoring Market Research

Over the last month I ve performed some market research to better understand the potential for co2mon.nz and to help me decide whether the product I ve built has a fit with the market or not. The key conclusions I ve drawn from this work are: Keep reading to hear more about the results that lead to those conclusions.

Survey The first piece of research I undertook was a survey covering three topics: views on indoor air quality, how respondents currently monitor indoor air quality and the desired features, including price, for a CO2 monitor. The survey was distributed to my extended personal network via social media, email and word of mouth. I offered respondents the opportunity to win a year of free monitoring as an incentive and received just under 70 responses overall - the lucky winner of that prize was Sam H of Auckland whose shiny new CO2 monitor will be in the mail shortly.

Views on indoor air quality
  • Nearly all respondents strongly agreed that clean, fresh indoor air is important for avoiding sickness and enabling our best work, learning and general cognitive performance, with not a single negative response.
  • 25% of respondents indicated they did not have a good understanding of the quality of the indoor air they were breathing versus 43% who indicated they had a good understanding of their indoor air quality.
  • Nearly 70% of respondents agreed (and greater than 40% strongly agreed) that real-time monitoring is beneficial and worth investing time and money in providing, with a similar distribution of responses agreeing it should be required in all shared indoor spaces.

Current ventilation monitoring approaches
  • For the home setting, using our senses was the most common method of understanding air quality, and only 6% of respondents were unhappy with their ability to monitor ventilation at home.
  • At work, trusting the owner of the building to monitor ventilation was the most common method, although using our senses and some personally collected data also featured for 20% of respondents. While the majority of respondents saw some room for improvement here, less than 20% of respondents were unsatisfied with the ability to monitor ventilation at work.
  • In shared public spaces using our senses and trusting the owner were equally popular with very little use of any data reported. The majority of respondents (40%) were unsatisfied with this situation with 34% seeing some room for improvement and very few being satisfied overall.

CO2 monitoring product features
  • A screen and WiFi were both strongly supported features with less than 10% of respondents seeing them as irrelevant and a large majority of answers skewing towards essential.
  • Coloured lights providing a quick indication were not viewed as important by 13% of respondents and while the majority of answers were towards essential there was also a large (22%) set of respondents who were indifferent to this feature.
  • The ability to access measurements and reports via a web interface was very mixed. Around 20% of respondents reported the feature as irrelevant, 20% essential with the majority seeing it as useful but mot essential.
  • Almost all respondents strongly indicated that additional air quality metrics beyond CO2 were important to collect.
  • Respondents mostly indicated the proposed prices are too high (64%), with essentially no responses suggesting they were too low and the balance (43%) in the middle. Only 5% of respondents indicated a preference for a rental option over a straight purchase.

Advertising In parallel with the survey, I worked with my cousin who runs a marketing agency, The Asset, to place some Facebook ads aiming to systematically evaluate what combination of images and text would draw the best response. It s been an interesting process - despite working for Google for 15 years, I know relatively little about the day to day practice of online advertising! I think we re about 50% of the way through that process of systematically building a funnel of traffic, it s been a steep learning curve and its clear there s significantly more thought and time that would need to be invested into this were it to be the primary driver of sales for a business. It s interested to see how what resonates or doesn t resonate with the audience is often completely different to what I expect, confirming the importance of having a process to evaluate and tweak how the advertising runs. After just under 2 weeks of advertising with a daily budget in the $20 - $30 range, my ads have had just under 17k impressions by 10k distinct people resulting in 76 visits to the co2mon.nz website, and zero sales. The ads themselves received 233 clicks, so there s clearly a lot of room for further improvement and revision of the ad text itself to present a more compelling message. Unfortunately the most common response and feedback to the ads themselves has been comments arguing that CO2 is wonderful, climate change is invented and all our problems would be solved if we had more CO2 everywhere. Tedious to deal with, but also useful reminder about awareness and interest in the problem to contrast with the results from the survey of my extended personal network!

Feedback from other conversations In addition to the survey and advertising I ve had conversations with some local air conditioning and ventilation businesses as well as a commercial building management firm - all providing similar feedback to the results from the survey - acknowledgement that air quality is important and relatively immaturely measured currently, but low urgency or pain to change or remedy that situation. Another interesting point that s come up in conversations with various small business owners is what to do if or when the monitoring shows a ventilation problem? The obvious answer of opening the windows more does not seem to be particularly well received. Without a compelling solution to offer to the potential problem that the monitoring might reveal I often sense a reluctance from people to invest too much time and money in something which may create a problem in a space they don t currently see as urgent.

Conclusions The responses are interesting and surprising to me a in a few ways (no interest in rental, favouring web interface over app), but at the end of the day lead to the two conclusions described above: Air quality is acknowledged as important, but monitoring it is not an urgent or pressing problem for most people. At home and work the majority of people are OK with relying on their senses or trusting someone else to maintain ventilation. They wouldn t object to improvements, but the feedback is that ventilation monitoring is not a problem people are actively looking to solve. The number of people who do see this as an urgent enough problem to invest money into solving is low - even within the biased sample of my extended network. There is a stronger set of evidence for the problem being seen as more urgent by the users of shared public spaces - but I ve not been able to find any evidence that the owners and managers of those spaces feel the same urgency or duty of care towards their users to invest in this space. Most of the opportunity is in the hardware rather than the software service. This signal comes through in the feedback on the pricing (preferring outright purchase vs rental), but it s also been directly expressed in the free-form comments and other conversations I ve had and the the relative importance given to the physical product features over the web/app interfaces in the survey results.

Wrap Up I m glad I finally spent the time doing this research, particularly the survey, these are good lessons to learn, even if I should have taken the time to learn them a year ago - so I can write that reminder (do your research before building a product) down as a key outcome of this process too! Stay tuned for more details on the other work I ve been doing recently on the hardware side of co2mon.nz and what these results mean for my overall plans. As always, I d love to hear from you if these results give you ideas or questions you d like to discuss.

2 June 2023

Matt Brown: Calling time on DNSSEC: The costs exceed the benefits

I m calling time on DNSSEC. Last week, prompted by a change in my DNS hosting setup, I began removing it from the few personal zones I had signed. Then this Monday the .nz ccTLD experienced a multi-day availability incident triggered by the annual DNSSEC key rotation process. This incident broke several of my unsigned zones, which led me to say very unkind things about DNSSEC on Mastodon and now I feel compelled to more completely explain my thinking: For almost all domains and use-cases, the costs and risks of deploying DNSSEC outweigh the benefits it provides. Don t bother signing your zones. The .nz incident, while topical, is not the motivation or the trigger for this conclusion. Had it been a novel incident, it would still have been annoying, but novel incidents are how we learn so I have a small tolerance for them. The problem with DNSSEC is precisely that this incident was not novel, just the latest in a long and growing list. It s a clear pattern. DNSSEC is complex and risky to deploy. Choosing to sign your zone will almost inevitably mean that you will experience lower availability for your domain over time than if you leave it unsigned. Even if you have a team of DNS experts maintaining your zone and DNS infrastructure, the risk of routine operational tasks triggering a loss of availability (unrelated to any attempted attacks that DNSSEC may thwart) is very high - almost guaranteed to occur. Worse, because of the nature of DNS and DNSSEC these incidents will tend to be prolonged and out of your control to remediate in a timely fashion. The only benefit you get in return for accepting this almost certain reduction in availability is trust in the integrity of the DNS data a subset of your users (those who validate DNSSEC) receive. Trusted DNS data that is then used to communicate across an untrusted network layer. An untrusted network layer which you are almost certainly protecting with TLS which provides a more comprehensive and trustworthy set of security guarantees than DNSSEC is capable of, and provides those guarantees to all your users regardless of whether they are validating DNSSEC or not. In summary, in our modern world where TLS is ubiquitous, DNSSEC provides only a thin layer of redundant protection on top of the comprehensive guarantees provided by TLS, but adds significant operational complexity, cost and a high likelihood of lowered availability. In an ideal world, where the deployment cost of DNSSEC and the risk of DNSSEC-induced outages were both low, it would absolutely be desirable to have that redundancy in our layers of protection. In the real world, given the DNSSEC protocol we have today, the choice to avoid its complexity and rely on TLS alone is not at all painful or risky to make as the operator of an online service. In fact, it s the prudent choice that will result in better overall security outcomes for your users. Ignore DNSSEC and invest the time and resources you would have spent deploying it improving your TLS key and certificate management. Ironically, the one use-case where I think a valid counter-argument for this position can be made is TLDs (including ccTLDs such as .nz). Despite its many failings, DNSSEC is an Internet Standard, and as infrastructure providers, TLDs have an obligation to enable its use. Unfortunately this means that everyone has to bear the costs, complexities and availability risks that DNSSEC burdens these operators with. We can t avoid that fact, but we can avoid creating further costs, complexities and risks by choosing not to deploy DNSSEC on the rest of our non-TLD zones.

But DNSSEC will save us from the evil CA ecosystem! Historically, the strongest motivation for DNSSEC has not been the direct security benefits themselves (which as explained above are minimal compared to what TLS provides), but in the new capabilities and use-cases that could be enabled if DNS were able to provide integrity and trusted data to applications. Specifically, the promise of DNS-based Authentication of Named Entities (DANE) is that with DNSSEC we can be free of the X.509 certificate authority ecosystem and along with it the expensive certificate issuance racket and dubious trust properties that have long been its most distinguishing features. Ten years ago this was an extremely compelling proposition with significant potential to improve the Internet. That potential has gone unfulfilled. Instead of maturing as deployments progressed and associated operational experience was gained, DNSSEC has been beset by the discovery of issue after issue. Each of these has necessitated further changes and additions to the protocol, increasing complexity and deployment cost. For many zones, including significant zones like google.com (where I led the attempt to evaluate and deploy DNSSEC in the mid 2010s), it is simply infeasible to deploy the protocol at all, let alone in a reliable and dependable manner. While DNSSEC maturation and deployment has been languishing, the TLS ecosystem has been steadily and impressively improving. Thanks to the efforts of many individuals and companies, although still founded on the use of a set of root certificate authorities, the TLS and CA ecosystem today features transparency, validation and multi-party accountability that comprehensively build trust in the ability to depend and rely upon the security guarantees that TLS provides. When you use TLS today, you benefit from:
  • Free/cheap issuance from a number of different certificate authorities.
  • Regular, automated issuance/renewal via the ACME protocol.
  • Visibility into who has issued certificates for your domain and when through Certificate Transparency logs.
  • Confidence that certificates issued without certificate transparency (and therefore lacking an SCT) will not be accepted by the leading modern browsers.
  • The use of modern cryptographic protocols as a baseline, with a plausible and compelling story for how these can be steadily and promptly updated over time.
DNSSEC with DANE can match the TLS ecosystem on the first benefit (up front price) and perhaps makes the second benefit moot, but has no ability to match any of the other transparency and accountability measures that today s TLS ecosystem offers. If your ZSK is stolen, or a parent zone is compromised or coerced, validly signed TLSA records for a forged certificate can be produced and spoofed to users under attack with minimal chances of detection. Finally, in terms of overall trust in the roots of the system, the CA/Browser forum requirements continue to improve the accountability and transparency of TLS certificate authorities, significantly reducing the ability for any single actor (say a nefarious government) to subvert the system. The DNS root has a well established transparent multi-party system for establishing trust in the DNSSEC root itself, but at the TLD level, almost intentionally thanks to the hierarchical nature of DNS, DNSSEC has multiple single points of control (or coercion) which exist outside of any formal system of transparency or accountability. We ve moved from DANE being a potential improvement in security over TLS when it was first proposed, to being a definite regression from what TLS provides today. That s not to say that TLS is perfect, but given where we re at, we ll get a better security return from further investment and improvements in the TLS ecosystem than we will from trying to fix DNSSEC.

But TLS is not ubiquitous for non-HTTP applications The arguments above are most compelling when applied to the web-based HTTP-oriented ecosystem which has driven most of the TLS improvements we ve seen to date. Non-HTTP protocols are lagging in adoption of many of the improvements and best practices TLS has on the web. Some claim this need to provide a solution for non-HTTP, non-web applications provides a motivation to continue pushing DNSSEC deployment. I disagree, I think it provides a motivation to instead double-down on moving those applications to TLS. TLS as the new TCP. The problem is that costs of deploying and operating DNSSEC are largely fixed regardless of how many protocols you are intending to protect with it, and worse, the negative side-effects of DNSSEC deployment can and will easily spill over to affect zones and protocols that don t want or need DNSSEC s protection. To justify continued DNSSEC deployment and operation in this context means using a smaller set of benefits (just for the non-HTTP applications) to justify the already high costs of deploying DNSSEC itself, plus the cost of the risk that DNSSEC poses to the reliability to your websites. I don t see how that equation can ever balance, particularly when you evaluate it against the much lower costs of just turning on TLS for the rest of your non-HTTP protocols instead of deploying DNSSEC. MTA-STS is a worked example of how this can be achieved. If you re still not convinced, consider that even DNS itself is considering moving to TLS (via DoT and DoH) in order to add the confidentiality/privacy attributes the protocol currently lacks. I m not a huge fan of the latency implications of these approaches, but the ongoing discussion shows that clever solutions and mitigations for that may exist. DoT/DoH solve distinct problems from DNSSEC and in principle should be used in combination with it, but in a world where DNS itself is relying on TLS and therefore has eliminated the majority of spoofing and cache poisoning attacks through DoT/DoH deployment the benefit side of the DNSSEC equation gets smaller and smaller still while the costs remain the same.

OK, but better software or more careful operations can reduce DNSSEC s cost Some see the current DNSSEC costs simply as teething problems that will reduce as the software and tooling matures to provide more automation of the risky processes and operational teams learn from their mistakes or opt to simply transfer the risk by outsourcing the management and complexity to larger providers to take care of. I don t find these arguments compelling. We ve already had 15+ years to develop improved software for DNSSEC without success. What s changed that we should expect a better outcome this year or next? Nothing. Even if we did have better software or outsourced operations, the approach is still only hiding the costs behind automation or transferring the risk to another organisation. That may appear to work in the short-term, but eventually when the time comes to upgrade the software, migrate between providers or change registrars the debt will come due and incidents will occur. The problem is the complexity of the protocol itself. No amount of software improvement or outsourcing addresses that. After 15+ years of trying, I think it s worth considering that combining cryptography, caching and distributed consensus, some of the most fundamental and complex computer science problems, into a slow-moving and hard to evolve low-level infrastructure protocol while appropriately balancing security, performance and reliability appears to be beyond our collective ability. That doesn t have to be the end of the world, the improvements achieved in the TLS ecosystem over the same time frame provide a positive counter example - perhaps DNSSEC is simply focusing our attention at the wrong layer of the stack. Ideally secure DNS data would be something we could have, but if the complexity of DNSSEC is the price we have to pay to achieve it, I m out. I would rather opt to remain with the simpler yet insecure DNS protocol and compensate for its short comings at higher transport or application layers where experience shows we are able to more rapidly improve and develop our security capabilities.

Summing up For the vast majority of domains and use-cases there is simply no net benefit to deploying DNSSEC in 2023. I d even go so far as to say that if you ve already signed your zones, you should (carefully) move them back to being unsigned - you ll reduce the complexity of your operating environment and lower your risk of availability loss triggered by DNS. Your users will thank you. The threats that DNSSEC defends against are already amply defended by the now mature and still improving TLS ecosystem at the application layer, and investing in further improvements here carries far more return than deployment of DNSSEC. For TLDs, like .nz whose outage triggered this post, DNSSEC is not going anywhere and investment in mitigating its complexities and risks is an unfortunate burden that must be shouldered. While the full incident report of what went wrong with .nz is not yet available, the interim report already hints at some useful insights. It is important that InternetNZ publishes a full and comprehensive review so that the full set of learnings and improvements this incident can provide can be fully realised by .nz and other TLD operators stuck with the unenviable task of trying to safely operate DNSSEC.

Postscript After taking a few days to draft and edit this post, I ve just stumbled across a presentation from the well respected Geoff Huston at last weeks RIPE86 meeting. I ve only had time to skim the slides (video here) - they don t seem to disagree with my thinking regarding the futility of the current state of DNSSEC, but also contain some interesting ideas for what it might take for DNSSEC to become a compelling proposition. Probably worth a read/watch!

18 April 2023

Matt Brown: co2mon.nz: Ventilation monitoring as a service

Previously, I explained why ventilation monitoring is important, and the opportunity I see to help accelerate deployment of high quality ventilation monitoring for small businesses and organisations. In this post, I m going to discuss my plans to tackle that opportunity:

My journey to ventilation monitoring I started looking into ventilation monitoring in detail last year when I wanted to ensure that the classrooms of our local primary school were well ventilated during the Omicron outbreak. That research revealed that the existing products on offer were challenging to deploy in a school environment from a cost perspective while also not providing perfect functionality. I built and deployed a set of monitors to our local school, along with a corresponding set of software and web services that provide management of the monitors and visibility into the collected data and trends. The benefit of the monitoring is evident - CO2 concentration drops immediately after a notification occurs, indicating the intended action of increasing ventilation is taking place. Initially this occurred more frequently, but over time low concentrations of CO2, indicating good ventilation levels, are regularly achieved across the school day thanks to simple changes in routine being established, such as leaving windows cracked open. As the benefits of the project to the school became clear, my ambition and vision for ventilation monitoring grew from wanting to see my children learn in a safe and effective environment to desiring safe and effective learning, working, living and social spaces for everyone.

co2mon.nz: Ventilation monitoring as a service prototype co2mon.nz is the service that I have launched to explore this opportunity. Based on the same hardware and software that I used for the school deployment I am selling both a physical CO2 monitor and an ongoing subscription service for the management and monitoring. Together, the monitor and software provide an effective and affordable path to understanding indoor air quality and ventilation status that anyone can use.

CO2 monitor The CO2 monitor hardware and firmware are based on Oliver Seiler s design using a Sensirion SCD40 module for CO2 measurement and providing immediate in-room feedback via an on-monitor screen as well as a red/orange/green traffic light . In addition to the immediate feedback provided by the monitor itself, a WiFi connection can be configured allowing every measurement to be reported back to the supporting monitoring and management service. At the current prototype stage, I am building all the hardware, 3D printing the plastic cases and assembling the circuit board from purchased components through a production line in my home workshop. The monitor is powered via a standard USB cable and wall adaptor for simplicity. The first batch of monitors I built have been operating smoothly for nearly a year now. Further refinement and development of the design is required but already I am impressed with the potential this combination of easily accessible components provides to enable widespread adoption and deployment of ventilation monitoring.

Monitoring and management service The web service provides a historical record of CO2 levels from each monitor through a set of graphs and other dashboards and allows for this data to be shared or published. This is useful both for providing visibility into ventilation status to those not directly present in the space (e.g. across an entire school) and to allow insights into trends and patterns in ventilation to be observed and acted upon. When I began the project I approached the software side as an opportunity to explore the real-world equivalents of many of the internal tools and components that I was used to using within Google. That s been a fun process, even though it means I ve invested far more time on the software than the current number of customers or business state can really justify. Communication with the monitors occurs via MQTT into a custom Go service responsible for maintaining the fleet and exposing the measurement data to Prometheus which acts as the storage layer. A second service (based on the Buffalo web framework) is responsible for serving the web interface containing the dashboards and configuration options. Alongside these core components another small Golang service manages a private X509 certificate authority to provide monitor identity and encryption of the measurement data and an automated pipeline of scripts and triggers takes care of firmware provisioning, sensor calibration and burn-in/failure testing when building batches of monitors. While simple, the design is easy to manage and could scale to support many, many more monitors without significant effort. I m hosting the infrastructure on AWS, which I ve found pleasant and easy to use, and while not a typical choice, I m very pleased with the performance and flexibility that Prometheus offers as a data storage layer.

Areas of development The prototype has been effective in proving the feasibility of offering ventilation monitoring as a service, but to evolve the prototype into an actual business is going to require further effort in the following areas:

Finding product market fit I m confident that what co2mon.nz is offering could provide a useful and needed service for many small business owners and numerous others, but the question of whether it s sufficiently compelling that a sizeable group of customers see enough value that they re willing to regularly pay for it and therefore sustain a business providing the service is not yet proven. While initial feedback on the idea has been positive, much of it has been from family and friends and I ve spent more time building the software (which has been fun and educational) than I have talking to actual potential customers and understanding their needs and wants. This is a classic trap that technical founders fall into, and I m guilty despite my best intentions it s comfortable and easy for me to work on the parts where I have existing experience and neglect putting time into the customer-facing sales, marketing and research which are more uncomfortable, new and challenging!

Scaling monitor production Scaling from the current prototype batches I have built to even modest levels (e.g. hundreds) of sales will require outsourcing. There are several potential New Zealand and overseas providers I can work with which look very promising, but will require iteration of the hardware design (e.g. to remove through-hole components which add complexity and expense to the PCB assembly process). The lead times I ve been quoted for working with these outsourced providers are in the order of 1-2 months per batch, and on top of this there are a number of other risks in terms of component availability and regulatory compliance that may add additional delays or time to resolve, particularly as my experience leans towards the software rather than the hardware. It seems counter-intuitive to worry about how to scale to meet demand that doesn t yet provably exist, but given the potentially long lead times involved in establishing outsourced production I think it s worth taking some risk in starting this journey early.

Next steps My aim now is to intentionally change my focus from the software to learning about the market and finding the right product to offer via two channels:
  1. Via online marketing, to reach a wide range of potential customers and allow rapid iteration and testing of a variety of product/service combinations, including referral options for existing customers.
  2. In-person with small businesses in my local area to seek out direct feedback and thoughts on the general issue of ventilation monitoring and the specific service I m offering.
One concrete possibility I m keen to test via both channels is a simple monthly price for the combined service, rather than purchasing a monitor and separate software subscription. This has the potential to be both simpler to market and explain to customers, while also opening the possibility of short-term rental or evaluation to customers who are not willing to make a long-term investment or commitment. In parallel to the customer-focused discovery work, but as an explicit lower priority, I ll order an initial small batch of outsourced monitors to begin gaining experience with that process accepting the risk that the time and cost may turn out to be wasted if the product is wrong or not a good match for customers needs.

3 April 2023

Matt Brown: Retrospective: Mar 2023

The key decision I made mid-March was to commit to pursuing ventilation monitoring as my primary product development focus. Prior to that decision, I hoped to use my writing plan to drive a breadth-first survey of the opportunities for each of my product ideas before deciding which had the best business potential to focus on first. Two factors changed my mind:
  1. As noted last month, I m finding the writing process much slower and harder than I expected the survey across all the ideas may not complete until mid-year or later!
  2. I ve realised that having begun building co2mon.nz last year, to stop work on the project at this point would leave me feeling that I had not done justice to developing the product and testing the market - seeing it to a conclusion is important to me.
This decision is an explicit choice to prioritize seeing a project through to a conclusion (successful, or otherwise) regardless of whether or not it has the highest potential of the various ideas I could invest time into. I m comfortable making that trade-off in this instance, but I am going to bound my time investment to two months. I ll evaluate at the end of May whether I m seeing sufficient traction and potential to justify continuing further with the idea. I had only one fully uninterrupted work week in March due to a combination of days out due to school trips, two LandSAR call outs and various farm maintenance tasks. April will be similarly disrupted given school holidays and a planned family trip to Brisbane. Sharpening my focus feels particularly necessary given this reality to ensure I m not spread overly thin.

Goal Scoring See last month s retrospective for a refresher on my scoring methodology.

Consulting - 4/10 Goal: Execute a series of successful consulting engagements, building a reputation for myself and leaving happy customers willing to provide testimonials that support a pipeline of future opportunities. Consulting hours were down from February, hitting only 31% of target this month as the client didn t make use of all the hours I had allocated for them. I didn t invest any time in advertising my services or developing new clients or projects over the month, which will now become a priority for April.

Product Development - 4/10 Goal: Grow my product development skill set by taking several ideas to MVP stage with customer feedback received, and launch at least one product which generates revenue and has growth potential. With the new focus entirely on co2mon.nz, I spent a lot of time re-working and developing my thinking around how I want to take this forward, specifically trying to analyse where I saw an opportunity in the market. After attending a workshop on finding product market fit using quantifiable metrics at the Southern SaaS conference this month, I ve realised that much of the time I spent on this analysis is too insular and focused on my own observations - I need to get out and talk to a lot more people and get more feedback on their needs and understanding of the space instead. Seems obvious in retrospect! I also spent a few days beginning to build another batch of prototype CO2 monitors so I have some units to use for experimentation and testing with potential customers as I get out and have those conversations. I can probably build one or two more batches of prototype monitors before needing to look at PCB assembly in earnest.

Professional Network Development - 8/10 Goal: To build a professional relationship with at least 30 new people this year. This goal continues to be my highlight with 8 new contacts added this month and catch-ups with 4 existing people I had not spoken to for a while. I joined the KiwiSaaS central community and attended the SouthernSaas conference this month as well, which has been time well spent given the workshop learnings discussed above.

Writing - 3/10 Goal: To publish a high-quality piece of writing on this site at least once a week. I published a single post, the first half of my updated ventilation monitoring business plan. I continue to find the writing process much harder and slower than I hoped or expected and remain well below my target publishing rate, but one post is better than zero! I tested working with an editor I contracted via UpWork who provided some very useful feedback on the structure of my writing which helped to unblock some of my progress. I plan to continue doing this for at least a few more posts.

Community - 5/10 Goal: To support the growth of my local technical community by volunteering my experience and knowledge with others through activities such as mentoring, conference talks and similar.

Feedback As always, I d love to hear from you if you have thoughts or feedback triggered by anything I ve written above.

28 March 2023

Matt Brown: Ventilation Monitoring: Ensuring every space has clean, fresh air

The importance of clean, fresh indoor air is one of the most tangible takeaways of the Covid-19 pandemic. In addition to being an effective risk mitigation strategy for reducing the spread of respiratory illnesses, clean, fresh air is necessary to enable effective cognitive performance. Monitoring indoor air quality is relatively easy to do, but traditionally has not been a key focus. I believe air quality monitoring should be accessible for any indoor space, and for highly occupied indoor spaces should be provided on a continuous basis. This post explores the need and an opportunity for a business that can accelerate the adoption of ventilation monitoring through the following topics:

The importance of indoor air quality Clean, fresh air is fundamental to life and health. That might sound obvious, but unfortunately being obvious is not enough to ensure the air we breathe is in fact always clean and healthy. Repeated studies have revealed that in many cases the air you re breathing at school, in the bus or at work and probably also at home falls well below the ideal of what clean, fresh air should be. Unclean air has potential long-term health impacts and has also been shown to lower cognitive performance impacting the ability to learn and work as well as increasing the risk of transmission of respiratory illnesses like Covid-19 and the flu. Ventilation (replacing old stale air with clean fresh air) is the most effective and economical method of improving and maintaining high indoor air quality. Most New Zealand buildings (including schools and houses) are designed to rely on manual ventilation (opening windows), while newer buildings, often including larger or commercial buildings may use mechanical ventilation involving fans and ducts. Mechanical ventilation including filtration may also be required in situations where the outdoor air is not clean and fresh such as in a city or next to a busy intersection.

Observing the invisible Overall air quality is a complex topic involving many contributing factors, many of which are invisible and not perceptible to us until well after adverse effects or irritation occur. This complexity and lack of visible signal is a large contributing factor to the ignorance and lack of attention towards indoor air quality that is prevalent in most buildings and indoor spaces today. Our attention is biased towards the risks that we can see, and this default bias has not been helped by hesitation and resistance to the idea that aerosol transmission and air quality is an important factor in preventing disease transmission that has only recently started to change. Zeynep Tufekci has a great overview that provides fascinating context for how an overreaction to the early incorrect theories of bad air and miasma causing disease contributed to aerosol transmission and air quality being incorrectly neglected for so long. Correcting this history of inattention to indoor air quality is going to take time and effort, but one significant step that we can take to help start the journey to ensuring all indoor spaces have clean, healthy air is to make the invisible part visible. The concentration of carbon dioxide (CO2) in a space is an incredibly effective and easy to measure proxy for the ventilation of a space. The atmospheric background level of CO2 is around 420 parts per million (ppm), while our exhaled breath has concentrations as high as 40,000 ppm. Without effective ventilation, one or more people breathing in an enclosed space will rapidly lead to an observable increase in CO2 concentration, which in turn provides a signal that the ventilation is insufficient and needs to be improved. Monitoring CO2 and improving ventilation is not a panacea for all possible air quality issues, but for the majority of buildings and indoor spaces, using CO2 as a proxy for ventilation and increasing ventilation when CO2 levels rise above recommended levels is a simple, effective and achievable approach that will deliver improvements in cognitive performance and reduction in the risk of disease transmission with few, if any, downsides or risks. See this Public Health Communication Centre briefing for a more detailed explanation.

Adding clean air to our hygiene practices We have well established expectations of hygiene for the food we eat and the water we drink and these expectations are codified in regulations that ensure those providing these services do so in a way that gives us confidence that we re not going to be at risk of illness. You may recall seeing food grade ratings prominently displayed on the walls of restaurants and cafes that you visit as an example of this. Why should the air we breathe be treated any differently? I think there is a strong argument that indoor air quality deserves regulation, both of the absolute quality of the air and ensuring that the practices and achieved air quality are clearly advertised and available. Ventilation monitoring via measurement of CO2 concentration provides an effective and achievable method that can be used to achieve this, and countries like Belgium and Japan are already starting to regulate indoor air quality. In the UK, the independent SAGE group of scientists has published Scores on the Doors , a proposal which demonstrates how CO2 monitoring can be helpful in providing information about the air quality of indoor spaces. Unfortunately there is no movement in any of these directions in New Zealand yet, and no sign that regulation or even a basic campaign to raise awareness of ventilation and air quality is even being planned. This is disappointing, but even if such work was planned, it would still require appropriate ventilation monitoring products and services to enable it, and while there are some options available, it is not a fully solved space yet.

Existing ventilation monitoring options Until recently the available offerings for ventilation monitoring have sat at two distinct ends of the price and quality spectrum:
  • Handheld air quality meters advertised as measuring CO2, but in reality reporting only an approximation. These meters do not contain actual CO2 sensors, and only approximate CO2 levels based on measurements of other components of the air. While cheap (often less than $100), these meters are not useful for providing reliable data that can be systematically used to assess and improve ventilation and should be avoided.
  • High-end building management systems (BMS), and industrial measurement products targeted at large buildings such as offices or commercial applications such as food production. These systems require specialist installation, often integrated with large whole-building air conditioning systems. These systems, if appropriately configured, can be a great solution for the types of buildings and spaces that can afford them, but by their nature and cost, they do not offer a solution for the majority of smaller buildings and indoor spaces where we tend to spend a lot of our time.
Over the last few years a growing number of companies have developed products that fit in between the unreliable air quality meters and the expensive BMS/industrial measurement products. Promising NZ-based options in this space include Air Suite, Tether and Monkeytronics. These products are wall mountable, resemble a smoke alarm and utilise a WiFi network to report their measurements to a supporting web service. Pricing varies between $200 and $300 ex GST per unit. Aranet, while not NZ based, provides a handheld monitor the Aranet4 Home, which is well regarded for quality and accuracy. Aranet4 Home devices are the most expensive in this space, retailing at $386 ex GST and offer a clunkier and less convenient set of connectivity options via a Bluetooth connection to an associated phone. To obtain similar reporting functionality to the other products requires upgrading to their Pro model and purchasing a separate base station at a combined cost of $1255 ex GST. Outside the commercial product offerings are a number of open source DIY options, which can be built by anyone with basic electronics knowledge. AirGradient is a leading example based in Thailand, and within New Zealand Oliver Seiler s CO2 Monitor provides similar functionality. These open source options have a parts cost in the $100-$150 range, depending on volume built and provide high-quality measurements via trusted CO2 sensors while also offering huge flexibility in terms of how they operate, interact with users and potential supporting web services.

An opportunity: Small businesses and organisations While a growing number of high-quality CO2 monitors has the potential to help drive increased adoption of ventilation monitoring, the plethora of small businesses and organisations that own, operate and manage many of the indoor spaces we visit on a day-to-day basis do not appear to be well served by these existing products. To deploy ventilation monitoring a small business or organisation needs to first become aware of the need or demand for it, and then have a simple and easy path to acquire and install the monitor and access the data. Little to no marketing or demand generation appears to be targeted towards this market from the existing businesses and tellingly, several of the products are not directly available for sale, requiring interaction with a salesperson to purchase. This indicates a focus on selling to larger customers who have a campus or portfolio of buildings and will purchase in larger quantities than the typical small business or organisation will. Small businesses and organisations are likely to occupy smaller buildings and spaces where manual ventilation is the prevalent method of improving and maintaining air quality. Maintaining clean, fresh air via manual ventilation requires the occupants of the space to receive an obvious and straightforward signal when action (opening windows, etc) is required. While the products above all tend to provide some form of local feedback and display in the room, the indication provided and notification of when to take action is less obvious and prominent than would be ideal in a situation where manual ventilation is being relied upon. Informally testing this opportunity with family and friends running small businesses over the last few months has resulted in promising feedback. One particular success story was the discovery of a fresh air duct on the air conditioning unit in a small office that had never been connected to the outside air and was simply recirculating air from the ceiling space back into an office! The resulting stuffiness and poor air quality had been noticed, but without the clear indication from the CO2 monitor that the air conditioning was actually making things worse, rather than better, the underlying issue had not been understood. With the issue fixed and the duct now connected, that business is now enjoying much more productive and healthy working conditions.

Next steps Many small businesses and organisations are likely to have poor air quality and opportunities for improvement similar to the example above that are waiting to be found and fixed, and the existing products available are neither focused or ideal for the needs of this market. I have spent some time over the past six months building a basic CO2 monitoring service that I have used to deploy ventilation monitoring to our local school, and a few other local businesses. There are a number of challenges that still need to be addressed in order to scale the business up, but I think there is a reasonable chance that I can build a viable business that offers an attractive and useful solution that would accelerate the deployment of ventilation monitoring for small businesses and organisations. In an upcoming post, I will explain the foundations of the service that I have built to date, the challenges that need to be overcome and how I plan to evolve the service from the current prototype into a sustainable, bootstrapped business.

4 March 2023

Matt Brown: Retrospective: Feb 2023

February ended up being a very short work month as I made a last minute decision to travel to Adelaide for the first 2 weeks of the month to help my brother with some house renovations he was undertaking. I thought I might be able to keep up with some work and my writing goals in the evenings while I was there, but days of hard manual labour are such an unfamiliar routine for me that I didn t have any energy left to make good on that intention. The majority of my time and focus for the remaining one and half weeks of the month was catching up on the consulting work that I had pushed back while in Adelaide. So while it doesn t make for a thrilling first month to look back and report on, overall I m not unhappy with what I achieved given the time available. Next month, I hope to be able to report some more exciting progress on the product development front as well.

Monthly Scoring Rubric I m evaluating each goal using a 10 point scale based on execution velocity and risk level, rather than absolute success (which is what I will look at in the annual/mid-year review). If velocity is good and risk is low or well managed the score is high, if either the velocity is low, or risk is high then the score is low. E.g:
  • 10 - perfect execution with low-risk, on track for significantly overachieving the goal.
  • 7 - good execution with low or well managed risk, highly likely to achieve the goal.
  • 5 - execution and risk are OK, should achieve the goal if all goes well.
  • 3 - execution or risk have problems, goal is at risk.
  • 0 - stalled, with no obvious path to recovery or success.

Goals

Consulting - 6/10 Goal: Execute a series of successful consulting engagements, building a reputation for myself and leaving happy customers willing to provide testimonials that support a pipeline of future opportunities.
  • I have one active local engagement assisting a software team with migrating their application from a single to multi-region architecture.
  • Two promising international engagements which were close to starting both cancelled based on newly issued company policies freezing their staffing/outsourcing budgets due to the current economic climate.
I m happy with where this is at - I hit 90% of my target hours in February (taking into account 2 weeks off) and the feedback I m receiving is positive. The main risk is the future pipeline of engagements, particularly if the cancellations indicate a new pattern. I m not overly concerned yet, as all the opportunities to date have been from direct or referred contacts in my personal network, so there s plenty of potential to more actively solicit work to create a healthier pipeline.

Product Development - 3/10 Goal: Grow my product development skill set by taking several ideas to MVP stage with customer feedback received, and launch at least one product which generates revenue and has growth potential.
  • Accelerating electrification - I continued to keep up with industry news and added some interesting reports to my reading queue, but made no significant progress towards identifying a specific product opportunity.
  • Farm management SaaS - no activity or progress at all.
  • co2mon.nz - I put significant thought and planning into how to approach a second iteration of this product. I started writing and completed 80% of a post to communicate the revised business plan, but it s not ready for publication yet, and even if it was, the real work towards it would need to actually happen to score more points here.
I had high hopes to make at least some progress in all three areas in February, but it just didn t happen due to lack of time. The good news is that since the low score here is purely execution driven, there s no new risks or blockers that will hinder much better progress here in March.

Professional Network Development - 8/10 Goal: To build a professional relationship with at least 30 new people this year. This is off to a strong start, I made 4 brand new connections and re-established contact with 9 other existing people I d not talked to for a while. I ve found the conversations energising and challenging and I m looking forward to continuing to keep this up.

Writing - 2/10 Goal: To publish a high-quality piece of writing on this site at least once a week. Well off track as already noted. I am enjoying the writing process and I continue to find it useful in developing my thoughts and forcing me to challenge my assumptions, but coupling the writing process with the thinking/planning that is a prerequisite to get those benefits definitely makes my output a lot slower than I was expecting. The slower speed, combined with the obvious time constraints of this month are not a great doubly whammy to be starting with, but I think with some planning and preparation it should have been avoidable by having a backlog of pre-written content for use in weeks where I m on holiday or otherwise busy. It s worth noting that among all the useful feedback I received, this writing target was often called out as overly ambitious, or likely to be counterproductive to producing quality writing. The feedback makes sense - for now I m not planning to change the goal (I might at my 6-month review point), but I am going to be diligent about adhering to my quality standard, which in turn means I m choosing to accept missing a weekly post here and there and taking a lower score on the goal overall. I apologise if you ve been eagerly waiting for writing that never arrived over February!

Community - 5/10 Goal: To support the growth of my local technical community by volunteering my experience and knowledge with others through activities such as mentoring, conference talks and similar.
  • I was an invited participant of the monthly KiwiSRE meet-up which was discussing SRE team models, and in particular I was able to speak to my experiences as described in an old CRE blog post on this topic.
  • I joined the program committee for SREcon23 APAC which is scheduled for mid-June in Singapore. I also submitted two talk proposals of my own (not sharing the details for now, since the review process is intended to be blind) which I m hopeful might make the grade with my fellow PC members!

Feedback As always, I d love to hear from you if you have thoughts or feedback triggered by anything I ve written above. In particular, it would be useful to know whether you find this type of report interesting to read and/or what you d like to see added/removed or changed.

2 February 2023

Matt Brown: 2023 Writing Plan

To achieve my goal of publishing one high-quality piece of writing per week this year, I ve put together a draft writing plan and a few organisational notes. Please let me know what you think - what s missing? what would you like to read more/less of from me? I aim for each piece of writing to generate discussion, inspire further writing, and raise my visibility and profile with potential customers and peers. Some of the writing will be opinion, but I expect a majority of it will take a learning by teaching approach - aiming to explain and present useful information to the reader while helping me learn more!

Topic Backlog The majority of my writing is going to fit into 4 series, allowing me to plan out a set of posts and narrative rather than having to come up with something novel to write about every week. To start with for Feb, my aim is to get an initial post in each series out the door. Long-term, it s likely that the order of posts will reflect my work focus (e.g. if I m spending a few weeks deep-diving into a particular product idea then expect more writing on that), but I will try and maintain some variety across the different series as well. This backlog will be maintained as a living page at https://www.mattb.nz/w/queue. Thoughts on SRE This series of posts will be pitched primarily at potential consulting customers who want to understand how I approach the development and operations of distributed software systems. Initial topics to cover include:
  • What is SRE? My philosophy on how it relates to DevOps, Platform Engineering and various other hot terms.
  • How SRE scales up and down in size.
  • My approach to managing oncall responsibilities, toil and operational work.
  • How to grow an SRE team, including the common futility of SRE transformations .
  • Learning from incidents, postmortems, incident response, etc.
Business plan drafts I have an ever-growing list of potential software opportunities and products which I think would be fun to build, but which generally don t ever leave my head due to lack of time to develop the idea, or being unable to convince myself that there s a viable business case or market for it. I d like to start sharing some very rudimentary business plan sketches for some of these ideas as a way of getting some feedback on my assessment of their potential. Whether that s confirmation that it s not worth pursuing, an expression of interest in the product, or potential partnership/collaboration opportunities - anything is better than the idea just sitting in my head. Initial ideas include:
  • Business oriented Mastodon hosting.
  • PDF E-signing - e.g. A Docusign competitor, but with a local twist through RealMe or drivers license validation.
  • A framework to enable simple, performant per-tenant at-rest encryption for SaaS products - stop the data leaks.
Product development updates For any product ideas that show merit and develop into a project, and particularly for the existing product ideas I ve already committed to exploring, I plan to document my product investigation and market research findings as a way of structuring and driving my learning in the space. To start with this will involve:
  • A series of explanatory posts diving into how NZ s electricity system works with a particular focus on how operational data that will be critical to managing a more dynamic grid flows (or doesn t flow!) today, and what opportunities or needs exist for generating, managing or distributing data that might be solvable with a software system I could build.
  • A series of product reviews and deep dives into existing farm management software and platforms in use by NZ farmers today, looking at the functionality they provide, how they integrate and generally testing the anecdotal feedback I have to date that they re clunky, hard to use and not well integrated.
  • For co2mon.nz the focus will be less on market research and more on exploring potential distribution channels (e.g. direct advertising vs partnership with air conditioning suppliers) and pricing models (e.g. buy vs rent).
Debugging walk-throughs Being able to debug and fix a system that you re not intimately familiar with is valuable skill and something that I ve always enjoyed, but it s also a skill that I observe many engineers are uncomfortable with. There s a set of techniques and processes that I ve honed and developed over the years for doing this which I think make the task of debugging an unfamiliar system more approachable. The idea, is that each post will take a problem or situation I ve encountered, from the initial symptom or problem report and walk through the process of how to narrow down and identify the trigger or root cause of the behaviour. Along the way, discussing techniques used, their pros and cons. In addition to learning about the process of debugging itself, the aim is to illustrate lessons that can be applied when designing and building software systems that facilitate and improve our experiences in the operational stage of a systems lifecycle where debugging takes place. Miscellaneous topics In addition the regular series above, stand-alone posts on the other topics may include:
  • The pros/cons I see of bootstrapping a business vs taking VC or other funding.
  • Thoughts on remote work and hiring staff.
  • AI - a confessional on how I didn t think it would progress in my lifetime, but maybe I was wrong.
  • Reflections on 15 years at Google and thoughts on subsequent events since my departure.
  • AWS vs GCP. Fight! Or with less click-bait, a level-headed comparison of the pros/cons I see in each platform.

Logistics

Discussion and comments A large part of my motivation for writing regularly is to seek feedback and generate discussion on these topics. Typically this is done by including comment functionality within the website itself. I ve decided not to do this - on-site commenting creates extra infrastructure to maintain, and limits the visibility and breadth of discussion to existing readers and followers. To provide opportunities for comment and feedback I plan to share and post notification and summarised snippets of selected posts to various social media platforms. Links to these social media posts will be added to each piece of writing to provide a path for readers to engage and discuss further while enabling the discussion and visibility of the post to grow and extend beyond my direct followers and subscribers. My current thinking is that I ll distribute via the following platforms:
  • Mastodon @matt@mastodon.nz - every post.
  • Twitter @xleem - selected posts. I m trying to reduce Twitter usage in favour of Mastodon, but there s no denying that it s still where a significant number of people and discussions are happening.
  • LinkedIn - probably primarily for posts in the business plan series, and notable milestones in the product development process.
In all cases, my aim will be to post a short teaser or summary paragraph that poses an question or relays an interesting fact to give some immediate value and signal to readers as to whether they want to click through rather than simply spamming links into the feed.

Feedback In addition to social media discussion I also plan to add a direct feedback path, particularly for readers who don t have time or inclination to participate in written discussion, by providing a simple thumbs up/thumbs down feedback widget to the bottom of each post, including those delivered via RSS and email.

Organisation To enable subscription to subsets of my writing (particularly for places like Planet Debian, etc where the more business focused content is likely to be off-topic), I plan to place each post into a set of categories:
  • Business
  • Technology
  • General
In addition to the categories, I ll also use more free-form tags to group writing with linked themes or that falls within one of the series described above.

26 January 2023

Matt Brown: Goals for 2023

This is the second of a two-part post covering my goals for 2023. See the first part to understand the vision, mission and strategy driving these goals. I want to thank my friend Nat, and Will Larson whose annual reviews I ve always enjoyed reading for inspiring me to write these posts. I ve found the process articulating my motivations and goals very useful to clarify my thoughts and create tangible next steps. I m grateful for that in and of itself, but I also hope that by publishing this you too might find it interesting, and the additional public accountability it creates will be a positive encouragement to me.

2023 Goals My focus for 2023 is to bootstrap a business that I can use to build software that solves real problems (see the strategy from the previous post for more details on this). I m going to track this via three goals:
  1. Execute a series of successful consulting engagements, building a reputation for myself and leaving happy customers willing to provide testimonials that support a pipeline of future opportunities.
  2. Grow my product development skill set by taking several ideas to MVP stage with customer feedback received, and launch at least one product which generates revenue and has growth potential.
  3. Develop and maintain a broad professional network.

Consulting Based on my background and experience, I plan to target my consulting across three areas:
  1. Leadership - building and growing operationally focused software teams following SRE/devops principles. A typical engagement may involve helping a client establish a brand new SRE/devops practice, or to strengthen and mature the existing practices used to build and operate reliable software in their team(s).
  2. Architecture - applying deep technical expertise to the design of large software systems, particularly focusing on their reliability and operability. A typical engagement may involve design input and decision making support for key aspects of a new system, providing external review and analysis to improve an existing system, or delivering actionable, tactical next steps during or immediately after a reliability crisis.
  3. Technology Strategy - translating high-level business needs into a technical roadmap that provides understandable explanations of the value software can deliver in that context, and the iterative series of appropriately sized projects required to realise it. A typical customer for this would be a small to medium sized business outside of the software industry with a desire to use software in a transformative way to improve their business but who does not employ the necessary in-house expertise to lead that transition.

Product Development There are three, currently extremely high level, product ideas that I m excited to explore:
  1. Improve co-ordination of electricity resources to accelerate the electrification of NZ s energy demand and the transition to zero carbon grid. NZ has huge potential to be a world-leader in decarbonising energy use through electrification, but requires a massive transition to realise the benefits. Many of the challenges to that transition involve coordination of an order of magnitude more distributed energy resources (DER) in a much more dynamic and software-oriented manner than the electricity industry is traditionally experienced with. The concept of improving DER coordination is not novel, but our grid has unique characteristics that mean we re likely to need to build localised solutions. There is a strong match between my experience with large, high-reliability distributed software systems, and this need. With renewed motivation in the industry for rapid progress and many conversations and consultations still in their early stages this a very compelling space to explore with the intent of developing a more detailed product opportunity to pursue.
  2. Reduce agricultural emissions by making high performance farm management, including effortless compliance reporting, straightforward, fun and effective for busy farmers. NZ s commitments to reduce agricultural emissions (our largest single sector) place increased compliance and reporting burdens on busy farmers who don t want to report the same data multiple times to different regulators and authorities. In tandem, rising business costs and constraints drive a need for continuous improvements in efficiency, performance and farm management processes in order to remain profitable. This in turn drives increases in complexity and the volume of data that farmers must work with. Many industry organisations and associated software developers offer existing products aimed at addressing aspects of these problems, but anecdotal feedback indicates these are poorly integrated, piecemeal solutions that are often frustrating to use - a burden rather than a source of continuous improvement. It looks like there could be an opportunity for a delightful, comprehensive farm management and reporting system to disrupt the industry and help farmers run more profitable and sustainable farms while also reducing compliance costs and effort.
  3. Lower sickness rates and improve cognitive performance by enabling every indoor space to benefit from continuous ventilation monitoring and reporting. Indoor air quality is important in reducing disease transmission risk and promoting optimal cognitive performance, but despite the current pandemic temporarily raising its profile, a focus on indoor air quality generally remains under the radar for most people. One factor contributing to this is the lack of widely available systems for continuously monitoring and reporting on air quality. I built https://co2mon.nz/ to help address this problem in my children s school during 2022. I see potential to further grow this business through marketing and raising awareness of the value of ventilation monitoring in all indoor environments.
In addition to these mission aligned product ideas, I m also interested in exploring the creation of small to medium sized SaaS applications that deliver useful value by serving the needs of a specialised or niche business or industry. Even when not directly linked to the overall mission, the development and operation of products of this type can support the strategy. Each application adds direct revenue and also contributes to achieving better economies of scale in the many backend processes and infrastructure required to deliver secure, reliable and performant software systems.

Developing my professional network To help make this goal more actionable and measurable I will track 3 sub goals:
  1. To build a professional relationship with at least 30 new people this year, meaning that we ve met and had a decent conversation at least a couple of times and keep in touch at least every few months in some form.
  2. To publish a piece of writing on this site at least once a week, and for many of those to generate interesting conversations and feedback. I ll use this as an opportunity to explore product ideas, highlight my consulting expertise and generally contribute interesting technical content into the world.
  3. To support the growth of my local technical community by volunteering my experience and knowledge with others through activities such as mentoring, conference talks and similar.

Next Steps Over the coming weeks I ll write more about each of these topics - you can use the box in the sidebar (or on the front page, if you re on a phone) to be notified when I post new writing (there s also an RSS feed here, for the geeks). I d love to have your feedback and engagement on these goals too - please drop me an email with your thoughts or even book a meeting - it won t be a distraction to me, you ll be helping me meet my goal of developing and maintaining my network :)

Matt Brown: Vision, Mission and Strategy

This is the first of a two-part post, covering high-level thoughts around my motivations and vision. Make sure to check out the second part for my specific goals for 2023. A new year is upon us! My plan was to be 6 months into the journey of starting a business by this point. I made some very tentative progress towards that goal in 2022, registering a company and starting some consulting work, but on the whole I ve found it much harder than expected to gather the necessary energy to begin that journey in earnest.

Reflection I m excited about the next chapter of my career, so the fact that I ve been struggling to get started has been frustrating. The only upside is that the delay has given me plenty of time to reflect on the last few years and what I can learn from them and draw some lessons to help better manage and sustain my energy going forward.

Purpose A large part of what I ve realised is that I should have left Google years ago. It was a great place to work, and I m incredibly grateful for everything I learned and received during my time there. For years it was my dream job, but my happiness had been declining, and instead of taking the (relatively small) risk of leaving to the unknown, I tried several variations of team and role in the hope of restoring the dream. The reality is that a significant chunk of my motivation and energy comes from being able to link my work back to a bigger purpose that delivers concrete positive impact in the world. I felt that link through Google s mission to make information universally accessible and useful for the first 10-11 years, but for the latter 4-5 years my ability to see that link was tenuous at best and trying to push through the challenges presented without that link providing a reliable source of energy is what drove my unhappiness and led to needing a longer break to recharge. I expect the challenges of starting a business to be even greater than what I experienced at Google, so the lesson I m taking from this is that it s crucial for me to understand what the link between my work and the bigger purpose with concrete positive impact in the world that I m aiming to contribute to is.

Community The second factor that I ve slowly come to realise has been missing from my career in the last few years has been participation in a professional community and a variety of enriching interpersonal relationships. As much as I value and need this type of interaction, fostering and sustaining it unfortunately doesn t come naturally to me. Working remotely since 2016 and then taking a 9 month break out of the industry are not particularly helpful contributors to building and maintaining a wide network either! The lesson here is simply that I m going to need to push past my comfort zone in reaching out and introducing myself to a range of people in order to grow my professional network, and equally I need to be diligent and disciplined in making time to maintain and regularly connect with people whom I respect and find energising to interact with.

Personal Influences Lastly, I ve been reflecting on a set of principles that are important to me. These are not so much new lessons, more confirming to myself what I value moving forward. There are many things I could include here, but to keep it somewhat brief, the key influences on my thinking are:
  • Independence - I can t entirely explain why or where it comes from, but since the start of my professional career (which I consider to be my consulting/freelancing development during high school) I ve understood that I m far more motivated by building and growing my own business than I am by working for someone else. Working for myself has always felt like the default and sensible course - I m excited to get back to that.
  • Openness - Open is better than closed, in terms of software, business model and organisational processes. This continues to be a strong belief and something I want to uphold in my business endeavours. Competition should be based on superior technical quality or service, not artificial constraints or barriers to entry that lock customers and users into a single solution or market. Protocols and networks should be open for wide participation and easily accessible to new entrants and competition.
  • People first - This applies both to how we work with each other - respectfully, valuing diversity and with integrity, and to how we apply technology to our world - with consideration for all stakeholders it may affect and awareness of both the intended and potential unintended impacts.

Framework Using Vision, Mission and Strategy as a planning framework has worked quite well for me when building and growing teams over the years, so I plan to re-use it personally to help organise the above reflections into a hopefully cohesive plan than results in some useful 2023 goals.

Vision Software systems contribute direct and meaningful impact to solving real problems in our world. Each word has a fair bit of meaning behind it for me, so breaking it down a little bit:
  • software systems - excite me because software is eating the world and has significant potential to do good.
  • contribute - Software alone doesn t solve problems, and misapplied can easily make things worse. To contribute software needs to be designed intentionally and evaluated with an awareness of risks it could pose within the complex system that is our modern world.
  • direct and meaningful impact - I m not looking for broad outcomes like improving productivity or communication, which apply generally across many problems. I want to see software applied to solve specific blockers whose removal unlocks significant progress towards solving a problem.
  • real - as opposed to straightforward problems. The types of issue where the acknowledgement of it as a real problem often ends the sentence as it feels too big to tackle. Climate change and pandemic risk are examples of real problems. Decentralising finance or selling more widgets are not.
  • in our world - is mostly filler to round out the sentence nicely, but I do think we should probably sort out the mess we re making on our own planet before trying to colonise anywhere else.

Mission To lead the development and operation of software systems that deliver new opportunities for individuals, businesses and communities to solve the real problems in their community. Again breaking down the intent a little bit:
  • lead - having a meaningful impact on real problems is a big job. I won t succeed as a one man band. It will require building and growing a larger team.
  • development and operation - development is fun and necessary, but I also wanted to highlight that the ongoing operation and integration of those software systems into the broader social and human systems of our world is an equally important and ongoing need.
  • new opportunities - are important to drive and motivate investment in the adoption of technology. Building or operating a system that maintains the status quo is not motivating for me.
  • individuals, businesses and communities - aka everyone! But each of these groups (as examples, not specific) will have diverse roles, needs and interactions with the software which must be considered to ensure the system achieves the desired contribution and impact.
  • their community - refines the ambition from the vision to an achievable scope of action within which to execute the mission. We won t solve our problems by targeting one big global fix, but if we each participate in solving the problems in our community, collectively it will make a difference.

Strategy Build a sustainable business that provides a home and infrastructure to support a continuous cycle of development, validation and growth of software systems fulfilling the mission and vision above.
  • Accumulate meaningful impact via a portfolio of systems rather than one big bet.
  • Focus on opportunities that promote the decarbonisation of our economy (the most pressing problem our society faces), but not at the expense of ignoring compelling opportunities to contribute impact to other real problems also.
  • Favour the marathon over the sprint - while being first can be fun and convey benefits, it s often the fast-followers who learn from the initial mistakes and deliver lasting change and broader impact.
In keeping with the final bullet point, I aim to evaluate the strategy against a long-term view of success. What excites me about it is that it has the potential to provide structure and clarity for my work while also enabling many future paths - from operating a portfolio of micro-SaaS products that each solve real problems for a specific niche or community, or diving deep into a single compelling opportunity for a year or two, joining with others to partner on shared ventures or some combination of all three and other variations in between.

Your Thoughts I consider this a first draft, which I intend to revise and evolve further over the next 6-12 months. I don t plan major changes to the intent or underlying ideas, but finding the best words to express and convey that intent clearly is not something I expect to get right on the first take. I d love to have your feedback and engagement as I move forward with this strategy - please use the box in the sidebar (or on the front page, if you re on a phone) to be notified when I post new writing, drop me an email with your thoughts or even book a meeting to say hi and discuss something in detail.

Goals for 2023 Next up - check out part two of this post to see my goals for 2023.

14 January 2023

Matt Brown: Rebooting...

Hi! After nearly 7 years of dormancy, I m rebooting this website and have a goal to write regularly on a variety of topics going forward. More on that and my goals in a coming post For now, this is just a placeholder note to help double-check that everything on the new site is working as expected and the letters are flowing through the pipes in the right places.

Technical Details I ve migrated the site from Wordpress, to a fully static configuration using Hugo and TailwindCSS for help with styling. For now hosting is still on a Linode VM in Fremont, CA, but my plan is to move to a more specialized static hosting service with better CDN reach in the very near future. If you want to inspect the innards further, it s all at https://github.com/mattbnz/web-matt.

Still on the TODO list
  • Improve the hosting situation as noted above.
  • Integrate Bert Hubert s nice audience minutes analytics script.
  • Write up (or find) LinkedIn/Twitter/Mastodon integration scripts to automatically post updates when a new piece of writing appears on the site, to build/notify followers and improve the social reach. Ideally, the script would then also update the page here with links to the thread(s), so that readers can easily join/follow any resulting conversation on the platform of their choice. I m not planning to add any direct comment or feedback functinoality on the site itself.
  • Add a newsletter/subscription option for folks who don t use RSS and would prefer updates via email rather than a social feed.

31 December 2021

Russell Coker: Links December 2021

Wired magazine has many short documentary films on YouTube, this one about How Photography is Affecting Our Brains is particularly good [1]. Matt Blaze wrote an informative blog post about Faraday cages for phones [2]. It seems that the commercial shielded bags are all pretty good while doing it yourself with aluminium foil may get similar results or may get much worse results with no obvious difference in the quality of the wrapping. Aluminium foil doesn t protect that well and doesn t protect consistently. A metal biscuit tin performed quite well and consistently, so that s a cheap option for reducing signals. Umair Haque wrote an insightful article about the single word that describes most of the problems the world faces right now [3]. Forbes has an informative article about the early days of the Ford company when they doubled wages, it proves that they didn t do so to enable workders to afford cars but to avoid staff turnover (which is expensive) [4]. Also the Ford company had a fascistic approach to employees, controlling what they were allowed to do in their spare time if they wanted the bonus payment. The wages weren t doubled, there was a bonus payment that would double the salary if the employee was eligible for the bonus. One thing that Forbes gets wrong is that they claim that it was only having higher pay than other companies that provided a benefit and that a higher minimum wage wouldn t, the problem with that idea is that a higher minimum wage would discourage people from having multiple jobs and allow more families to not have the mother working (a condition for a man to get the Ford bonus was for his wife to not work). The WSJ has an interesting article about Intel s datacenter for running all the different configurations of CPUs that they have supported over the last 10 years for security tests [5]. My Thinkpad (which is less than 10yo) is vulnerable to one of the SPECTRE family of exploits as Intel hasn t released microcode to fix it, getting fixed microcode out for all the systems from major vendors like Lenovo would be a good idea if they want to improve their security. NPR has an interesting article about the correlation between support for Trump in counties of the US with lack of vaccination and Covid19 deaths [6]. No surprises, but it s good to see the graphs. Cory Doctorow wrote an interesting article on the lack of slack in the current American education system [7]. It s not that bad in Australia but we are unfortunately moving in the American direction. Teen Vogue has an insightful article about the problems with the focus on resilience [8], while resilience is good we should make it a higher priority to avoid putting people in situations where they need to be resiliant than on encouraging resilience.

17 August 2014

Matt Brown: GPG Key Transition

Firstly, thanks to all who responded to my previous rant. It turns out exactly what I wanted does exist in the form of a ID-000 format smartcard combined with a USB reader. Perfect. No idea why I couldn t find that on my own prior to ranting, but very happy to have found it now. Secondly, now that I ve got my keys and management practices in order, it is time to begin transitioning to my new key. Click this link to find the properly signed, full transition statement. I m not going to paste the full statement into this post, but my new key is:
pub   4096R/A48F065A 2014-07-27 [expires: 2016-07-26]
      Key fingerprint = DBB7 64A1 797D 2E21 C799  3CD7 A628 CB5F A48F 065A
uid                  Matthew Brown <matt @mattb.net.nz>
uid                  Matthew Brown <mattb @debian.org>
sub   4096R/1937883F 2014-07-27 [expires: 2016-07-26]
If you signed my old key, I d very much appreciate a signature on my new key, details and instructions in the transition statement. I m happy to reciprocate if you have a similarly signed transition statement to present.

12 July 2014

Matt Brown: GPG Key Management Rant

2014 and it s still annoyingly hard to find a reasonable GPG key management system for personal use All I want is to keep the key material isolated from any Internet connected host, without requiring me to jump through major inconvenience every time I want to use the key. An HSM/Smartcard of some sort is an obvious choice, but they all suck in their own ways:
* FSFE smartcard it s a smartcard, requires a reader, which are generally not particular portable compared to a USB stick.
* Yubikey Neo restricted to 2048 bits, doesn t allow imports of primary keys (only subkeys), so you either generate on device and have no backup, or maintain some off-device primary key with only subkeys on the Neo, negating the main benefits of it in the first place.
* Smartcard HSM similar problems to the Neo, plus not really supported by GPG well (needs 2.0 with specific supporting module version requirements).
* Cryptostick made by some Germans, sounds potentially great, but perpetually out of stock. Which leaves basically only the roll your own dm-crypt+LUKS usb stick approach. It obviously works well, and is what I currently use, but it s a bunch of effort to maintain, particularly if you decide, as I have, that the master key material can never touch a machine with a network connection. The implication is that you now need to keep an airgapped machine around, and maintain a set of subkeys that are OK for use on network connected machines to avoid going mad playing sneakernet for every package upload. The ideal device would be a USB form factor, supporting import of 4096 bit keys, across all GPG capabilities, but with all crypto ops happening on-device, so the key material never leaves the stick once imported. Ideally also cheap enough (e.g. ~100ish currency units) that I can acquire two for redundancy. As far as I can tell, such a device does not exist on this planet. It s almost enough to make a man give up on Debian and go live a life of peace and solitude with the remaining 99.9% of the world who don t know or care about this overly complicated mess of encryption we ve wrought for ourselves. end rant.

16 March 2012

Matt Brown: Kindle Reading Stats

I ve written before about my initial investigations into the Kindle, and I ve learnt much more about the software and how it communicates with the Amazon servers since then, but it all requires detailed technical explanation which I can never seem to find the motivation to write down. Extracting reading data out of the system log files is however comparatively simple. I m a big fan of measurement and data so my motivation and goal for the Kindle log files was to see if I could extract some useful information about my Kindle use and reading patterns. In particular, I m interested in tracking my pace of reading, and how much time I spend reading over time. You ll recall from the previous post that the Kindle keeps a fairly detailed syslog containing many events, including power state changes, and changes in the Booklet software system including opening and closing books and position information. You can eyeball any one of those logfiles and understand what is going on fairly quickly, so the analysis scripts are at the core just a set of regexps to extract the relevant lines and a small bit of logic to link them together and calculate time spent in each state/book. You can find the scripts on Github: https://github.com/mattbnz/kindle-utils Of course, they re not quite that simple. The Kindle doesn t seem to have a proper hardware clock (or mine has a broken hardware clock). My Kindle comes back from every reboot thinking it s either at the epoch or somewhere in the middle of 2010, the time doesn t get corrected until it can find a network connection and ping an Amazon server for an update, so if you have the network disabled it might be many days or weeks of reading before the system time is updated to reality. Once it has a network connection it uses the MCC reported by the 3G modem to infer what timezone it should be in, and switches the system clock to local time. Unfortunately the log entries all look like this:
110703:193542 cvm[7908]: I TimezoneService:MCCChanged:mcc=310,old=GB,new=US:
110703:193542 cvm[7908]: I TimezoneService:TimeZoneChange:offset=-25200,zone=America/Los_Angeles,country=US:
110703:193542 cvm[7908]: I LipcService:EventArrived:source=com.lab126.wan,name=localTimeOffsetChanged,arg0=-25200,arg1=1309689302:
110703:193542 cvm[7908]: I TimezoneService:LTOChanged:time=1309689302000,lto=-25200000:
110703:183542 system: I wancontrol:pc:processing "pppstart"
110703:193542 cvm[7908]: I LipcService:EventArrived:source=com.lab126.wan,name=dataStateChanged,arg0=2,arg1=:
110703:183542 cvm[7908]: I ConnectionService:LipcEventArrived:source=com.lab126.cmd,name=intfPropertiesChanged,arg0=
,arg1=wan:
110703:183542 cvm[7908]: W ConnectionService:UnhandledLipcEvent:event=intfPropertiesChanged:
110703:193542 wifid[2486]: I wmgr:event:handleWpasupNotify(<2>CTRL-EVENT-DISCONNECTED), state=Searching:
110703:113542 wifid[2486]: I spectator:conn-assoc-fail:t=374931.469106, bssid=00:00:00:00:00:00:
110703:113542 wifid[2486]: I sysev:dispatch:code=Conn failed:
110703:183542 cvm[7908]: I LipcService:EventArrived:source=com.lab126.wifid,name=cmConnectionFailed,arg0=Failed to connect to WiFi network,arg1=
:
Notice how there is no timezone information associated with the date/time information on each line. Worse still the different daemons are logging in at least 3 different timezones/DST offsets all interspersed within the same logfile. Argh!! So our simple script that just extracts a few regexps and links them together nearly doubles in size to handle the various time and date convolutions that the logs present. Really, the world should just use UTC everywhere. Life would be so much simpler. The end result is a script that spits out information like:

B000FC1PJI: Quicksilver: Read 1 times. Last Finished: Fri Mar 16 18:30:57 2012
- Tue Feb 21 11:06:24 2012 => Fri Mar 16 18:30:57 2012. Reading time 19 hours, 29 mins (p9 => p914)
... Read 51 books in total. 9 days, 2 hours, 29 mins of reading time
I haven t got to the point of actually calculating reading pace yet, but the necessary data is all there and I find the overall reading time stats interesting enough for now. If you have a jailbroken Kindle, I d love for you to have a play and let me know what you think. You ll probably find logs going back at least 2-3 weeks still on your Kindle to start with, and you can use the fetch-logs script to regularly pull them down to more permanent storage if you desire.

24 November 2011

Matt Brown: How I m voting in 2011

It s general election time again in New Zealand this year, with the added twist of an additional referendum on whether to keep MMP as our electoral system. If you re not interested in New Zealand politics, then you should definitely skip the rest of this post. I ve never understood why some people consider their voting choices a matter of national security, so when via Andrew McMillan, I saw a good rationale for why you should share your opinion I found my excuse to write this post. Party Vote
I ll be voting for National. I m philosophically much closer to National than Labour, particularly on economic and personal responsibility issues, but even if I wasn t the thought of having Phil Goff as Prime Minister would be enough to put me off voting Labour. His early career seems strong, but lately it s been one misstep and half-truth after another, the remainder of the Labour caucus and their likely support partners don t offer much reassurance either. If I was left-leaning and the mess that Labour is in wasn t enough to push me over to National this year then I d vote Greens and hope they saw the light and decided to partner with National. Electorate Vote
I live in Dublin, but you stay registered in the last electorate where you resided, which for me is Tamaki. I have no idea who the candidates there are, so I ll just be voting for the National candidate for the reasons above. MMP Referendum
I have no real objections to MMP and I think it s done a good job of increasing representation in our parliament. I like that parties can bring in some star players without them having to spend time in an electorate. I don t like the tendency towards unstable coalitions that our past MMP results have sometimes provided. Of the alternatives, STV is the only one that I think should be seriously considered, FPP and it s close cousin SM don t give the proportionality of MMP and PV just seems like a simplified version of STV with limited other benefit. If you re going to do preferential voting, you might as well do it properly and use STV. So, I ll vote for a change to STV, not because I m convinced that MMP is wrong, but because I think it doesn t hurt for the country to spend a bit more time and energy confirming that we have the right electoral system. If the referendum succeeds and we get another referendum between MMP and something other than STV in 2014, I ll vote to keep MMP. If we have a vote between MMP and STV in 2014 I m not yet sure how I d vote. STV is arguably an excellent system, but I worry that it s too complex for most voters to understand. PS. Just found this handy list of 10 positive reasons to vote for National, if you re still undecided and need a further nudge. Kiwiblog: 10 positive reasons to vote National

13 June 2011

Jonathan Wiltshire: StartSSL: finally, a trustworthy certifier*

Matt Brown writes about StartCom, the Israeli issuer providing basic SSL certificates for nothing. In fact I ve been using StartSSL certificates for about three years now, but I get them issued to Level 2 verification which incurs a fee. (It s more expensive now than when I was first validated, but still good value.) StartCom are the only issuer I ve ever dealt with who work like this. They validate the individual, using: This makes me trust them far more than other issuers, who don t bother with any meaningful validation at all. Their approach is to establish identity, then allow you to: Although this doesn t make up for trust (the presence of an SSL certificate doesn t guarantee the data you send is safe upon arrival) it does make me much happier to see a CA taking proper verification measures instead of just handing out certificates at random and it s much cheaper for me too, being verified once and then issuing as many certificates as I need. Highly recommended. *that is, more trustworthy
Comments
StartSSL: finally, a trustworthy certifier* is a post from: jwiltshire.org.uk Flattr

12 June 2011

Matt Brown: Using StartCom Free SSL certificates with Cyrus imapd

A stumbled across Start Com a few months ago, an Israeli company that run a Certificate Authority (CA) called StartSSL with a root certificate in all the modern browsers and operating systems. Best of all they don t participate in the cartel run by the rest of the SSL certificate industry and offer domain validated certificates at the price it costs them to issue them nothing. I had the first opportunity to use their services today when I needed an SSL cert to secure the IMAP server I run for my parents and I was very pleased with the experience. The web interface is a bit weird and you have to jump through some strange hoops, but to save paying more money to the SSL certificate cartel it seemed more than worthwhile. Like most CAs these days the certificate which signs your server certificate is not the actual root certificate included in your operating system or browser, but an intermediate CA certificate which is in turn signed by the root certificate. This means that you have to ensure that your server includes the intermediate CA certificate alongside the server certificate so the client can validate the entire path back to the root. Unlike Apache which explicitly allows you to specify a certificate chain file, the openssl methods used by Cyrus 2.2 only seem to recognise a single CA certificate in the file pointed to by tls_ca_file. All as not lost however, as the openssl libraries are actually quite smart and will automagically determine which intermediate certs they need to bundle into the handshake if you install them appropriately under /etc/ssl/certs (at least on Debian). The trick is that you have to install the intermediate CA cert into a file named after the hash of the certificate, like so:

# wget http://www.startssl.com/certs/sub.class1.server.ca.pem -O /etc/ssl/certs/startcom-class1-intermediate.pem
# hash=$(openssl x509 -hash -noout -in /etc/ssl/certs/startcom-class1-intermediate.pem)
# ln -s ./startcom-class1-intermediate.pem /etc/ssl/certs/$ hash .0
# ls -l /etc/ssl/certs/$ hash .0
lrwxrwxrwx 1 root root 34 2011-06-13 07:43 /etc/ssl/certs/ea59305e.0 -> ./startcom-class1-intermediate.pem
Then in imapd.conf:

tls_cert_file: /etc/ssl/certs/your-server-cert.pem
tls_key_file: /etc/ssl/private/your-server-key.key
tls_ca_file: /etc/ssl/certs/startcom-ca.pem
Voila. Works everywhere I ve tried so far. Start Com Highly Recommended. I ll be using them for any future SSL certificate purchases (e.g. EV certs) that I need to make.

11 May 2011

Matt Brown: Linux ignores IPv6 router advertisements when forwarding is enabled

IPv6 adoption is increasing, and along with it come a new set of behaviours and defaults that system administrators and users must learn and become familiar with. I was recently caught out by Linux s handling of IPv6 router advertisements (RAs) when forwarding is also enabled on the interface. It took me a while to figure out and searching for obvious terms (such as those in the first half of the title of this post) didn t immediately yield useful answers, so here is my attempt to help shed some light on the subject. By default Linux will ignore IPv6 RAs if the interface is configured to forward traffic. This is in line with RFC2462 which states that a device should be either a Host or a Router. If you re forwarding packets you re a router and you re therefore expected to be sending RAs, not receiving them. This policy does make a certain amount of sense but there are obviously situations where it can be useful to accept RAs and still forward packets over the interface[0]. The confusing part is that the Linux IPv6 stack allows the accept_ra sysctl to be set to 1 (enabled) at the same time as the forwarding sysctl is set to 1, yet all incoming RAs are ignored with no hint as to why. If you re not aware that the default behaviour is to ignore RAs when forwarding is enabled it looks very much like autoconfiguration has simply broken. The key piece of information is that makes everything as clear as mud is realising that the forwarding and accept_ra sysctl s are not simple boolean enabled/disabled flags like many of their brethren. There are instead three possible values for each, all clearly documented in sysctl.txt, when you take the time to read it. Ironically the documentation states the type of the values as BOOLEAN even though they re not at least it helped me to feel better about my hasty assumption that the sysctl s were boolean values.
accept_ra BOOLEAN
Accept Router Advertisements; autoconfigure using them. Possible values are:
0 Do not accept Router Advertisements.
1 Accept Router Advertisements if forwarding is disabled.
2 Overrule forwarding behaviour. Accept Router Advertisements
even if forwarding is enabled. Functional default: enabled if local forwarding is disabled.
disabled if local forwarding is enabled.
The documentation for forwarding is similar, but much longer, so you can refer to the link above to see it. Conclusion: If you want to autoconfigure IPv6 addresses on an interface that you re also forwarding IPv6 traffic over, you need to set accept_ra to 2. No doubt there are more IPv6 quirks and defaults like this waiting to trap me in the future :) [0] Arguably you really don t want to be autoconfiguring addresses on your router ever, but that s a philosophical debate that isn t really relevant to this post.

Next.