Sync port forward context managed hangs on exit for around a minute

[jacobtomlinson]

Which project are you reporting a bug for?

kr8s

What happened?

While investigating #569 I noticed a hang locally when running a sync port forward context manager. It seems to take a long time to exit the context manager.

# Create web server Pod
kubectl run webserver --image nginx --port 80
kubectl expose po webserver

import kr8s
import requests
import time

[svc] = kr8s.get("svc", "webserver")
start = time.time()
with svc.portforward(remote_port=80, local_port="auto") as local_port:
    print(f"After enter: {time.time() - start:.1}s")
    resp = requests.get(f"http://localhost:{local_port}/")
    print(f"After rerquest: {time.time() - start:.1}s")
print(f"After exit: {time.time() - start:.1}s")

After enter: 0.001s
After rerquest: 0.04s
After exit: 70s

# Clean up
kubectl delete svc/webserver po/webserver

Anything else?

No response

[jacobtomlinson]

This does not happen in the asyncio implementation.

In [2]: import kr8s
   ...: import httpx
   ...: import time
   ...: 
   ...: [svc] = [s async for s in kr8s.asyncio.get("svc", "webserver")]
   ...: start = time.time()
   ...: async with svc.portforward(remote_port=80, local_port="auto") as local_port, httpx.AsyncClient() as client:
   ...:     print(f"After enter: {time.time() - start:.1}s")
   ...:     resp = await client.get(f"http://localhost:{local_port}/")
   ...:     print(f"After rerquest: {time.time() - start:.1}s")
   ...: print(f"After exit: {time.time() - start:.1}s")
After enter: 0.01s
After rerquest: 0.05s
After exit: 0.05s

[miguelgarcia]

Hey @jacobtomlinson I did some debugging and realized that the issue is not on kr8s, but on how requests honors HTTP connection keep alive mechanism. I believe the right thing to do when you want to close the connection after the request is completed is to use the connection HTTP header, as follows:

import kr8s
from kr8s.objects import Service
import requests
import time

[svc] = kr8s.get("svc", "webserver")
start = time.time()
with svc.portforward(remote_port=80, local_port="auto") as local_port:
    print(f"After enter: {time.time() - start:.1}s")
    resp = requests.get(f"http://localhost:{local_port}/", headers={"Connection": "close"})
    print(f"After request: {time.time() - start:.1}s")
print(f"After exit: {time.time() - start:.1}s")

In this case I get:

After enter: 0.0004s
After request: 0.03s
After exit: 0.03s

[jacobtomlinson]

Thanks for digging into this @miguelgarcia! That makes a lot of sense. Requests holds the HTTP connection open and our port forward waits for this to close before closing.

I agree that telling requests to close the connection is "correct" but I wonder whether we want to forcibly close open connections when we get to the end of our context manager.

[miguelgarcia]

I think that closing connections when the context manager exits makes sense. Perhaps there should be some optinal argument that can be passed to portforward() to make it wait for connections to close, but by default forcibly close them.

[miguelgarcia]

Adding more info. asyncio recently added Server.close_clients() and Server.abort_clients() available since Python 3.13. Perhaps we should leverage one of those methods when available