0

I use asyncio to write to mongo, using motor library.

When I have few bulk_writes, it works with no problem.

However, when I have many write in the same time, I get an exception RuntimeError: can't start new thread.

File "/usr/local/lib/python3.7/site-packages/motor/metaprogramming.py", line 77, in method **unwrapped_kwargs)
File "/usr/local/lib/python3.7/site-packages/motor/frameworks/asyncio/__init__.py", line 74 in run_on_executor 
_EXECUTOR, functools.partial(fn, *args, **kwargs))
File "uvloop/loop.pyx", line 2702, in uvloop.loop.Loop.run_in_exector
File "/usr/local/lib/python3.7/concurrent/features/thread.py", line 160, in submit
   self._adjust_thread_count()
File "/usr/local/lib/python3.7/concurrent/features/thread.py", line 181, in _adjust_thread_count
   t.start()
File "usr/local/lib/python3.7/threading.py", line 847, in start
 _start_new_thread(self._bootsrap, ())
RuntimeError: can't start new thread

I tried to change maxPoolSize, but it didn't work.

Important facts:

  1. In my local computer, it works with no errors. However, in Openshift I have this problems.
  2. In Openshift I run my code via gunicorn via gunicorn app:app --worker-class uvicorn.workers.UvicornWorker
  3. In Openshift, when I have only one worker, it works. But with 2+ worker I have this problem.
  4. I don't open many connection of AsyncIOMontorClient, I have only two at a time.
  5. With pymongo with almost the same code, I have no error, but there is no asyncio support in pymongo.
  6. Without the part of mongo, my code works with no problems.
Ofir
  • 590
  • 9
  • 19
  • I presume you have checked https://stackoverflow.com/questions/63532917/python-cant-start-new-thread-multiprocessing, and https://stackoverflow.com/questions/32283003/python-cant-start-new-thread-100-active-threads and https://stackoverflow.com/questions/50368906/python-exception-in-threading-cant-start-new-thread – Paul Cornelius Jul 19 '22 at 09:29

1 Answers1

0

Solved. There is a limit of 1024 threads per openshift pod.

Ofir
  • 590
  • 9
  • 19