Author: ceridwen
I now have working test environments set up for null (no container,
build on the host system),
schroot,
and
qemu. After fixing some bugs,
null and qemu now pass all their tests!
schroot still has a permission error related to disorderfs. Since the
same code works for null and qemu and for schroot when disorderfs is
disabled, it's something specific to disorderfs and/or its combination
with schroot. The following is debug output that shows
ls for the
build directory on the testbed before and after the mock build, and
stat for both the build directory and the mock build artifact
itself. The first control run, without disorderfs, succeeds:
test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
total 20
drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
-rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
-rwxr--r-- 1 user user 175 Jun 3 15:42 mock_failure.py
-rw-r--r-- 1 user user 252 Jun 14 16:06 template.ini
-rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
test.py: DBG: testbed command exited with code 0
test.py: DBG: testbed command ['sh', '-ec', 'cd /tmp/autopkgtest.5oMipL/control/ ;\n python3 mock_build.py ;\n'], kind short, sout raw, serr pipe, env ['LANG=en_US.UTF-8', 'HOME=/nonexistent/first-build', 'VIRTUAL_ENV=~/code/reprotest/.tox/py35', 'PATH=~/code/reprotest/.tox/py35/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'PYTHONHASHSEED=559200286', 'TZ=GMT+12']
test.py: DBG: testbed command exited with code 0
test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
total 20
drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
-rw-r--r-- 1 root root 0 Jul 18 15:06 artifact
-rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
-rwxr--r-- 1 user user 175 Jun 3 15:42 mock_failure.py
-rw-r--r-- 1 user user 252 Jun 14 16:06 template.ini
-rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
test.py: DBG: testbed command exited with code 0
test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/control/'], kind short, sout raw, serr raw, env []
File: '/tmp/autopkgtest.5oMipL/control/'
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 56h/86d Inode: 1351634 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 1000/ user) Gid: ( 1000/ user)
Access: 2016-07-18 15:06:31.105915342 -0400
Modify: 2016-07-18 15:06:31.089915352 -0400
Change: 2016-07-18 15:06:31.089915352 -0400
Birth: -
test.py: DBG: testbed command exited with code 0
test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/control/artifact'], kind short, sout raw, serr raw, env []
File: '/tmp/autopkgtest.5oMipL/control/artifact'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: fc01h/64513d Inode: 40767795 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2016-07-18 15:06:31.089915352 -0400
Modify: 2016-07-18 15:06:31.089915352 -0400
Change: 2016-07-18 15:06:31.089915352 -0400
Birth: -
test.py: DBG: testbed command exited with code 0
test.py: DBG: sending command to testbed: copyup /tmp/autopkgtest.5oMipL/control/artifact /tmp/tmpw_mwks82/control_artifact
schroot: DBG: executing copyup /tmp/autopkgtest.5oMipL/control/artifact /tmp/tmpw_mwks82/control_artifact
schroot: DBG: copyup_shareddir: tb /tmp/autopkgtest.5oMipL/control/artifact host /tmp/tmpw_mwks82/control_artifact is_dir False downtmp_host /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52//tmp/autopkgtest.5oMipL
schroot: DBG: copyup_shareddir: tb(host) /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/control/artifact is not already at destination /tmp/tmpw_mwks82/control_artifact, copying
test.py: DBG: got reply from testbed: ok
That last bit indicates that copy command for the build artifact from
the testbed to a temporary directory on the host succeeded. This is the
debug output from the second run, with disorderfs enabled:
test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
total 20
drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
-rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
-rwxr--r-- 1 user user 175 Jun 3 15:42 mock_failure.py
-rw-r--r-- 1 user user 252 Jun 14 16:06 template.ini
-rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
test.py: DBG: testbed command exited with code 0
test.py: DBG: testbed command ['sh', '-ec', 'cd /tmp/autopkgtest.5oMipL/disorderfs/ ;\n umask 0002 ;\n linux64 --uname-2.6 python3 mock_build.py ;\n'], kind short, sout raw, serr pipe, env ['LC_ALL=fr_CH.UTF-8', 'CAPTURE_ENVIRONMENT=i_capture_the_environment', 'HOME=/nonexistent/second-build', 'VIRTUAL_ENV=~/code/reprotest/.tox/py35', 'PATH=~/code/reprotest/.tox/py35/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin/i_capture_the_path', 'LANG=fr_CH.UTF-8', 'PYTHONHASHSEED=559200286', 'TZ=GMT-14']
test.py: DBG: testbed command exited with code 0
test.py: DBG: testbed command ['ls', '-l', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
total 20
drwxr-xr-x 2 user user 4096 Jul 15 23:43 __pycache__
-rw-r--r-- 1 root root 0 Jul 18 15:06 artifact
-rwxr--r-- 1 user user 2340 Jun 28 18:43 mock_build.py
-rwxr--r-- 1 user user 175 Jun 3 15:42 mock_failure.py
-rw-r--r-- 1 user user 252 Jun 14 16:06 template.ini
-rwxr-xr-x 1 user user 1600 Jul 15 23:18 tests.py
test.py: DBG: testbed command exited with code 0
test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/disorderfs/'], kind short, sout raw, serr raw, env []
File: '/tmp/autopkgtest.5oMipL/disorderfs/'
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 58h/88d Inode: 1 Links: 3
Access: (0755/drwxr-xr-x) Uid: ( 1000/ user) Gid: ( 1000/ user)
Access: 2016-07-18 15:06:31.201915291 -0400
Modify: 2016-07-18 15:06:31.185915299 -0400
Change: 2016-07-18 15:06:31.185915299 -0400
Birth: -
test.py: DBG: testbed command exited with code 0
test.py: DBG: testbed command ['stat', '/tmp/autopkgtest.5oMipL/disorderfs/artifact'], kind short, sout raw, serr raw, env []
File: '/tmp/autopkgtest.5oMipL/disorderfs/artifact'
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 58h/88d Inode: 7 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2016-07-18 15:06:31.185915299 -0400
Modify: 2016-07-18 15:06:31.185915299 -0400
Change: 2016-07-18 15:06:31.185915299 -0400
Birth: -
test.py: DBG: testbed command exited with code 0
test.py: DBG: sending command to testbed: copyup /tmp/autopkgtest.5oMipL/disorderfs/artifact /tmp/tmpw_mwks82/experiment_artifact
schroot: DBG: executing copyup /tmp/autopkgtest.5oMipL/disorderfs/artifact /tmp/tmpw_mwks82/experiment_artifact
schroot: DBG: copyup_shareddir: tb /tmp/autopkgtest.5oMipL/disorderfs/artifact host /tmp/tmpw_mwks82/experiment_artifact is_dir False downtmp_host /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52//tmp/autopkgtest.5oMipL
schroot: DBG: copyup_shareddir: tb(host) /var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/disorderfs/artifact is not already at destination /tmp/tmpw_mwks82/experiment_artifact, copying
schroot: DBG: cleanup...
schroot: DBG: execute-timeout: schroot --run-session --quiet --directory=/ --chroot jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52 --user=root -- rm -rf -- /tmp/autopkgtest.5oMipL
rm: cannot remove '/tmp/autopkgtest.5oMipL/disorderfs': Device or resource busy
schroot: DBG: execute-timeout: schroot --quiet --end-session --chroot jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52
Unexpected error:
Traceback (most recent call last):
File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 708, in mainloop
command()
File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 646, in command
r = f(c, ce)
File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 584, in cmd_copyup
copyupdown(c, ce, True)
File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 469, in copyupdown
copyupdown_internal(ce[0], c[1:], upp)
File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 494, in copyupdown_internal
copyup_shareddir(sd[0], sd[1], dirsp, downtmp_host)
File "~/code/reprotest/reprotest/lib/VirtSubproc.py", line 408, in copyup_shareddir
shutil.copy(tb, host)
File "/usr/lib/python3.5/shutil.py", line 235, in copy
copyfile(src, dst, follow_symlinks=follow_symlinks)
File "/usr/lib/python3.5/shutil.py", line 114, in copyfile
with open(src, 'rb') as fsrc:
PermissionError: [Errno 13] Permission denied: '/var/lib/schroot/mount/jessie-amd64-ac94881d-ae71-4f24-a004-1847889d5d52/tmp/autopkgtest.5oMipL/disorderfs/artifact'
ls shows that the artifact is created in the right place. However,
when reprotest tries to copy it from the testbed to the host, it gets
a permission error. The traceback is coming from
virt/schroot, and
it's a Python
open() call that's failing. Note that the permissions
are wrong for the second run, but that's expected because my schroot
is stable so the umask bug isn't fixed yet; and that the
rm error
from disorderfs not being unmounted early enough (see below). I
expect to see the umask test fail, though, not a crash in every test
where the build succeeds.
After a great deal of effort, I isolated the bug that was causing the
process to hang not to my code or autopkgtest's code, but to
CPython and
contextlib. It's supposed to be
fixed in CPython 3.5.3, but for now I've worked around the problem by
monkey-patching the patch provided in the latter issue onto
contextlib.
Here is my current to-do list:
- Fix PyPi not installing the virt/ scripts correctly.
- Move the disorderfs unmount into the shell script. (When the virt/
scripts encounter an error, they try to delete a temporary
directory, which fails if disorderfs is mounted, so the script needs
to unmount it before that happens.)
- Find and fix the schroot/disorderfs permission error bug.
- Convert my notes on setting up for the tests into something useful
for users.
- Write scripts to synch version numbers and documentation.
- Fix the headers in the autopkgtest code to conform to reprotest
style.
- Add copyright information for the contextlib monkey-patch and the
autopkgtest files I've changed.
- Close
#829113
as wontfix.
And here are the questions I'd like to resolve before the second release:
- Is there any other documentation that's essential? Finishing the
documentation will come later.
- Should I release before finishing the rest of the variation? This
will slow down the release of the first version with something
resembling full fuctionality.
- Do I need to write a chroot test now? Given the duplication with
schroot, I'm unconvinced this is worthwhile.