This looks straightforward and is far from it. I expect tool support will
improve in the future. Meanwhile, this blog post serves as a step by step
explanation for what is going on in code that I'm about to push to my team.
Let's take this relatively straightforward python code. It has a function
printing an int, and a decorator that makes it argument optional, taking it
from a global default if missing:
from unittest import mock
default = 42
def with_default(f):
def wrapped(self, value=None):
if value is None:
value = default
return f(self, value)
return wrapped
class Fiddle:
@with_default
def print(self, value):
print("Answer:", value)
fiddle = Fiddle()
fiddle.print(12)
fiddle.print()
def mocked(self, value=None):
print("Mocked answer:", value)
with mock.patch.object(Fiddle, "print", autospec=True, side_effect=mocked):
fiddle.print(12)
fiddle.print()
It works nicely as expected:
$ python3 test0.py
Answer: 12
Answer: 42
Mocked answer: 12
Mocked answer: None
It lacks
functools.wraps and typing, though. Let's add them.
Adding functools.wraps
Adding a simple
@functools.wraps, mock unexpectedly stops working:
# python3 test1.py
Answer: 12
Answer: 42
Mocked answer: 12
Traceback (most recent call last):
File "/home/enrico/lavori/freexian/tt/test1.py", line 42, in <module>
fiddle.print()
File "<string>", line 2, in print
File "/usr/lib/python3.11/unittest/mock.py", line 186, in checksig
sig.bind(*args, **kwargs)
File "/usr/lib/python3.11/inspect.py", line 3211, in bind
return self._bind(args, kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/inspect.py", line 3126, in _bind
raise TypeError(msg) from None
TypeError: missing a required argument: 'value'
This is the new code, with explanations and a fix:
# Introduce functools
import functools
from unittest import mock
default = 42
def with_default(f):
@functools.wraps(f)
def wrapped(self, value=None):
if value is None:
value = default
return f(self, value)
# Fix:
# del wrapped.__wrapped__
return wrapped
class Fiddle:
@with_default
def print(self, value):
assert value is not None
print("Answer:", value)
fiddle = Fiddle()
fiddle.print(12)
fiddle.print()
def mocked(self, value=None):
print("Mocked answer:", value)
with mock.patch.object(Fiddle, "print", autospec=True, side_effect=mocked):
fiddle.print(12)
# mock's autospec uses inspect.getsignature, which follows __wrapped__ set
# by functools.wraps, which points to a wrong signature: the idea that
# value is optional is now lost
fiddle.print()
Adding typing
For simplicity, from now on let's change
Fiddle.print to match its wrapped signature:
# Give up with making value not optional, to simplify things :(
def print(self, value: int None = None) -> None:
assert value is not None
print("Answer:", value)
Typing with ParamSpec
# Introduce typing, try with ParamSpec
import functools
from typing import TYPE_CHECKING, ParamSpec, Callable
from unittest import mock
default = 42
P = ParamSpec("P")
def with_default(f: Callable[P, None]) -> Callable[P, None]:
# Using ParamSpec we forward arguments, but we cannot use them!
@functools.wraps(f)
def wrapped(self, value: int None = None) -> None:
if value is None:
value = default
return f(self, value)
return wrapped
class Fiddle:
@with_default
def print(self, value: int None = None) -> None:
assert value is not None
print("Answer:", value)
mypy complains inside the wrapper, because while we forward arguments we don't
constrain them, so we can't be sure there is a
value in there:
test2.py:17: error: Argument 2 has incompatible type "int"; expected "P.args" [arg-type]
test2.py:19: error: Incompatible return value type (got "_Wrapped[P, None, [Any, int None], None]", expected "Callable[P, None]") [return-value]
test2.py:19: note: "_Wrapped[P, None, [Any, int None], None].__call__" has type "Callable[[Arg(Any, 'self'), DefaultArg(int None, 'value')], None]"
Typing with Callable
We can use explicit Callable argument lists:
# Introduce typing, try with Callable
import functools
from typing import TYPE_CHECKING, Callable, TypeVar
from unittest import mock
default = 42
A = TypeVar("A")
# Callable cannot represent the fact that the argument is optional, so now mypy
# complains if we try to omit it
def with_default(f: Callable[[A, int None], None]) -> Callable[[A, int None], None]:
@functools.wraps(f)
def wrapped(self: A, value: int None = None) -> None:
if value is None:
value = default
return f(self, value)
return wrapped
class Fiddle:
@with_default
def print(self, value: int None = None) -> None:
assert value is not None
print("Answer:", value)
if TYPE_CHECKING:
reveal_type(Fiddle.print)
fiddle = Fiddle()
fiddle.print(12)
# !! Too few arguments for "print" of "Fiddle" [call-arg]
fiddle.print()
def mocked(self, value=None):
print("Mocked answer:", value)
with mock.patch.object(Fiddle, "print", autospec=True, side_effect=mocked):
fiddle.print(12)
fiddle.print()
Now mypy complains when we try to omit the optional argument, because Callable
cannot represent optional arguments:
test3.py:32: note: Revealed type is "def (test3.Fiddle, Union[builtins.int, None])"
test3.py:37: error: Too few arguments for "print" of "Fiddle" [call-arg]
test3.py:46: error: Too few arguments for "print" of "Fiddle" [call-arg]
says:
Callable cannot express complex signatures such as functions that take a
variadic number of arguments, overloaded functions, or functions that have
keyword-only parameters. However, these signatures can be expressed by
defining a Protocol class with a call() method:
Let's do that!
Typing with Protocol, take 1
# Introduce typing, try with Protocol
import functools
from typing import TYPE_CHECKING, Protocol, TypeVar, Generic, cast
from unittest import mock
default = 42
A = TypeVar("A", contravariant=True)
class Printer(Protocol, Generic[A]):
def __call__(_, self: A, value: int None = None) -> None:
...
def with_default(f: Printer[A]) -> Printer[A]:
@functools.wraps(f)
def wrapped(self: A, value: int None = None) -> None:
if value is None:
value = default
return f(self, value)
return cast(Printer, wrapped)
class Fiddle:
# function has a __get__ method to generated bound versions of itself
# the Printer protocol does not define it, so mypy is now unable to type
# the bound method correctly
@with_default
def print(self, value: int None = None) -> None:
assert value is not None
print("Answer:", value)
if TYPE_CHECKING:
reveal_type(Fiddle.print)
fiddle = Fiddle()
# !! Argument 1 to "__call__" of "Printer" has incompatible type "int"; expected "Fiddle"
fiddle.print(12)
fiddle.print()
def mocked(self, value=None):
print("Mocked answer:", value)
with mock.patch.object(Fiddle, "print", autospec=True, side_effect=mocked):
fiddle.print(12)
fiddle.print()
New mypy complaints:
test4.py:41: error: Argument 1 to "__call__" of "Printer" has incompatible type "int"; expected "Fiddle" [arg-type]
test4.py:42: error: Missing positional argument "self" in call to "__call__" of "Printer" [call-arg]
test4.py:50: error: Argument 1 to "__call__" of "Printer" has incompatible type "int"; expected "Fiddle" [arg-type]
test4.py:51: error: Missing positional argument "self" in call to "__call__" of "Printer" [call-arg]
What happens with class methods, is that the function object has a
__get__
method that generates a bound versions of itself. Our Printer protocol does not
define it, so mypy is now unable to type the bound method correctly.
Typing with Protocol, take 2
So... we add the function descriptor methos to our Protocol!
A lot of this is taken from
this discussion.
# Introduce typing, try with Protocol, harder!
import functools
from typing import TYPE_CHECKING, Protocol, TypeVar, Generic, cast, overload, Union
from unittest import mock
default = 42
A = TypeVar("A", contravariant=True)
# We now produce typing for the whole function descriptor protocol
#
# See https://github.com/python/typing/discussions/1040
class BoundPrinter(Protocol):
"""Protocol typing for bound printer methods."""
def __call__(_, value: int None = None) -> None:
"""Bound signature."""
class Printer(Protocol, Generic[A]):
"""Protocol typing for printer methods."""
# noqa annotations are overrides for flake8 being confused, giving either D418:
# Function/ Method decorated with @overload shouldn't contain a docstring
# or D105:
# Missing docstring in magic method
#
# F841 is for vulture being confused:
# unused variable 'objtype' (100% confidence)
@overload
def __get__( # noqa: D105
self, obj: A, objtype: type[A] None = None # noqa: F841
) -> BoundPrinter:
...
@overload
def __get__( # noqa: D105
self, obj: None, objtype: type[A] None = None # noqa: F841
) -> "Printer[A]":
...
def __get__(
self, obj: A None, objtype: type[A] None = None # noqa: F841
) -> Union[BoundPrinter, "Printer[A]"]:
"""Implement function descriptor protocol for class methods."""
def __call__(_, self: A, value: int None = None) -> None:
"""Unbound signature."""
def with_default(f: Printer[A]) -> Printer[A]:
@functools.wraps(f)
def wrapped(self: A, value: int None = None) -> None:
if value is None:
value = default
return f(self, value)
return cast(Printer, wrapped)
class Fiddle:
# function has a __get__ method to generated bound versions of itself
# the Printer protocol does not define it, so mypy is now unable to type
# the bound method correctly
@with_default
def print(self, value: int None = None) -> None:
assert value is not None
print("Answer:", value)
fiddle = Fiddle()
fiddle.print(12)
fiddle.print()
def mocked(self, value=None):
print("Mocked answer:", value)
with mock.patch.object(Fiddle, "print", autospec=True, side_effect=mocked):
fiddle.print(12)
fiddle.print()
It works! It's typed! And mypy is happy!
The topic of this posting already tells you that an old Unix guy tells
stories about old techniques.
I'm a happy NIS (formerly YP) user since 30+ years. I started using it
with SunOS 4.0, later using it with Solaris and with Linux since 1999.
In the past, a colleague wasn't happyly using NIS+ when he couldn't
log in as root after a short time because of some well known bugs and wrong
configs. NIS+ was also much slower than my NIS setup. I know
organisations using NIS for more than 80.000 user accounts in 2024.
I know the security implications of NIS but I can live with them,
because I manage all computers in the network that have access to the
NIS maps. And NIS on Linux offers to use shadow maps, which are only
accessible to the root account. My users are forced to use very
long passwords.
Unfortunately NIS support for the PAM modules was removed in Debian
in pam 1.4.0-13, which means Debian 12 (bookworm) is
lacking NIS support in PAM, but otherwise it is still supported.
This only affects changing the NIS password via passwd. You can still
authenticate users and use other NIS maps.
But 
Dear Debian community,
this is Bits from DPL for October. In addition to a summary of my
recent activities, I aim to include newsworthy developments within
Debian that might be of interest to the broader community. I believe
this provides valuable insights and foster a sense of connection across
our diverse projects. Also, I welcome your feedback on the format and
focus of these Bits, as community input helps shape their value.
Ada Lovelace Day 2024
As outlined in my platform, I'm committed to increasing the diversity of
Debian developers. I hope the recent article celebrating Ada Lovelace
Day 2024 featuring interviews with 
The coffee shop Ibtehal and me used to visit during the midnight
Grating at a chemist shop in Mombasa, Kenya
Me with players at the Nairobi Gymkhana Club
Cricket pitch at the Nairobi Gymkhana Club
A view of the cricket ground inside the Nairobi Gymkhana Club
Scoreboard at the Nairobi Gymkhana cricket ground
A matatu with a Notorious BIG portrait.
Inside the Giraffe Center
SGR train at Nairobi Terminus.
Interior of the SGR train
Mombasa Terminus station.
The balcony at Tulia Backpackers Hostel had a nice view of the ocean.
A room inside the hostel with fans and mosquito nets on the beds
Nyali beach is a white sand beach
This is the ferry service for crossing the river.
Diani beach is a popular beach in Kenya. It is a white sand beach.
Selfie with truck drivers who gave me the free ride
Newspapers at a stand in Kenya covering news on the airport closure
Spanish Omelette served in breakfast at Nairobi Transit Hotel
Ugali with eggs.
Another short status update of what happened on my side last month. Besides a 
A hot-fix release 1.0.13-1, consisting of two small PRs relative to
the last regular 
Almost all of my Debian contributions this month were