6

I'm trying to understand why the limits have decided a task needs to be killed, and how it's doing the accounting. When my GCE Docker container kills a process, it shows something like:

Task in /404daacfcf6b9e55f71b3d7cac358f0dc921a2d580eed460c2826aea8e43f05e killed as a result of limit of /404daacfcf6b9e55f71b3d7cac358f0dc921a2d580eed460c2826aea8e43f05e 
memory: usage 2097152kB, limit 2097152kB, failcnt 74571 
memory+swap: usage 0kB, limit 18014398509481983kB, failcnt 0 
kmem: usage 0kB, limit 18014398509481983kB, failcnt 0 
Memory cgroup stats for /404daacfcf6b9e55f71b3d7cac358f0dc921a2d580eed460c2826aea8e43f05e: cache:368KB rss:2096784KB rss_huge:0KB mapped_file:0KB writeback:0KB inactive_anon:16KB active_anon:2097040KB inactive_file:60KB active_file:36KB unevictable:0KB 
[ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name 
[ 4343]     0  4343     5440       65      15        0             0 bash 
[ 4421]     0  4421   265895     6702      77        0             0 npm 
[ 4422]     0  4422    12446     2988      28        0             0 gunicorn 
[ 4557]     0  4557   739241   346035    1048        0             0 gunicorn 
[ 4560]     0  4560     1086       24       8        0             0 sh 
[ 4561]     0  4561     5466      103      15        0             0 bash 
[14594]     0 14594   387558   168790     672        0             0 node 
Memory cgroup out of memory: Kill process 4557 (gunicorn) score 662 or sacrifice child 
Killed process 4557 (gunicorn) total-vm:2956964kB, anon-rss:1384140kB, file-rss:0kB

Supposedly the memory hit a 2GB usage limit, and something needs to die. According to the cgroup stats, I appear to have 2GB of usage in active_anon and rss.

When I look at the table of process stats, I don't see where the 2GB is:

For rss, I see the two major processes 346035 + 168790 = 514MB? For total_vm, I see three major processes 265895 + 739241 + 387558 = 1.4GB?

But when it decides to kill the gunicorn process, it says it had 3GB of Total VM and 1.4GB of Anon RSS. I don't see how this follows from the above numbers at all...

For most of it's life, according to top, the gunicorn process appears to hum along with 555m RES and 2131m VIRT and 22% MEM * 2.5GB box = 550MB of memory usage. (I haven't yet been able to time it properly to peek at top values at the time it dies...)

  • Can someone help me understand this?
  • Under what accounting, do these sum to 2GB of usage? (virtual? rss? something else?)
  • Is there something else besides top/ps I should use to track how much memory a process is using for the purposes of docker's killing it?
Mike Lambert
  • 1,976
  • 1
  • 17
  • 31

1 Answers1

0

From what I know, the total_vm and rss are counted in 4kB (refer to: https://stackoverflow.com/a/43611576), instead of kB.

So for pid<4557>:

  • rss=346035, means anon-rss:1384140kB (=346035*4kB)
  • total_vm=739241, means total-vm:2956964kB(=739241*4kB)

This will explain your mem usage very well.

南北东篱
  • 1
  • 2