Skip to content

Bug: Reaper is force-removed at interpreter exit since 4.15.0, session containers are never reaped #1140

Open

Description

@pantafive

Describe the bug

Since 4.15.0, containers started by a process are left running after that process exits normally. Ryuk never reaps them.

#1017 registered Reaper.delete_instance with atexit to close the Ryuk socket before interpreter shutdown. delete_instance() also calls Reaper._container.stop(), which is remove(force=True), so the Ryuk container gets SIGKILL at the same moment the socket closes:

https://github.com/testcontainers/testcontainers-python/blob/6146aa1/src/testcontainers/core/container.py#L544

Ryuk only reaps a session RYUK_RECONNECTION_TIMEOUT (10s by default) after the last connection drops. Killed at exit, it never gets there. Every container of a session that ends normally without stopping its containers itself stays behind. This is the common case Ryuk exists for, e.g. a session-scoped pytest fixture that relies on Ryuk for cleanup.

docker events for the Ryuk container at interpreter exit:

start   <ryuk id>
kill    <ryuk id> sig=9
destroy <ryuk id>

On 4.14.2 the same script leaves nothing behind. With the process ending through os._exit(0), which skips atexit, 4.15.0 works too, and the Ryuk log shows the normal path:

Client disconnected: 172.17.0.1:49870
Timeout waiting for connection
Removed 1 container(s), 0 network(s), 0 volume(s) 0 image(s)

Expected: the process exits, Ryuk removes the session's containers about 10s later.
Actual: Ryuk is force-removed at exit, the session's containers keep running.

To Reproduce

python -c "
from testcontainers.core.container import DockerContainer
c = DockerContainer('alpine:3.20').with_command('sleep infinity').start()
print(c.get_wrapped_container().short_id)
"
sleep 15
docker ps --filter label=org.testcontainers.lang=python
# 4.14.2: empty. 4.15.0 and main: the alpine container is still Up.

Workarounds: pin testcontainers<4.15, or drop the handler after the reaper is created with atexit.unregister(Reaper.delete_instance).

Closing only the socket at exit keeps the fix from #1017 (no ResourceWarning) and lets Ryuk do its job. #1125 hits a 409 from this same atexit call when Ryuk is already being removed. Fixing this means the atexit path no longer runs into it. I'll open a PR with a fix and a regression test.

Runtime environment

  • testcontainers 4.15.0, reproduced on main at 6146aa1
  • Python 3.12.12
  • macOS with Colima (Ubuntu 24.04.4 VM, Docker Engine 29.2.1)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions