Skip to content

50MB Parent PID Issue

Jump to bottom
Richard Forth edited this page Jul 26, 2019 · 12 revisions

This explains the parent PID thing, and why the script aborts when it finds the parent PID of apache is greater than 50 MBs, and why it's important.

So, a couple of years ago I came across a server I was troubleshooting at my company, where apache2buddy was just giving some bizarre calculations, I even debugged all the code and verified there was nothing wrong in my logic.

Let me explain what we found:

This particular server had 12GB of RAM, now, at our company it is a standard practice to assign a 2GB swap partition, so even with 14GB of available "Memory", you too would be surprised to see an apache2buddy report telling me that somehow apache is currently using over 21 GB of RAM.

The server was running just fine, it was responsive to keyboard and commands, and wasnt swapping, oom killer wasnt kicking in, what was the issue?

I didn't know, as I say, my first thoughts were that it must be my script, so I spent a lot of time debugging and triple checking the calculations, I even went through and did all the steps apache2buddy would take, manually, and sure enough the maths came out exactly the same - I added up all of the RSS column totals for httpd processes, and it did indeed total over 21GB of RAM (this is, resident, in memory RAM) yet the server only had 12GB, what was going on?

After confabbing with my senior tech colleagues it became very clear what was happening:

inheritance

  1. The parent process was over 200MB in size (for apache that's pretty unusual) a normal parent PID is in the low teens of megabytes, on a production system, on new installs the apache parent PIDF can be consuming as little as 6 MB, one can only assume this particular customer had every module under the sun loaded at startup.

  2. In Prefork mode (the only real mode apache2buudy supports) apache forks child processes which serve the incoming requests.

  3. By the nature of a forked process, it takes on (inherits) all of the memory of the parent (though it may not actually be using it) and this is reflected in the RSS column of 'ps'. This is the data that apache2buddy.pl uses to calculate all of the real memory usage, and it turns out there's no real way to determine real from inherited memory, RSS is the figure to use.

Bloated parent, inherited by children:

Parent (200MB+ )
|-- Child (200MB+)
|-- Child (200MB+)
|-- Child (200MB+)
|-- Child (200MB+)
|-- Child (200MB+)

Here we can see all the children have inherited the size of the parent, and these calculations, can exceed the actual installed RAM and swap by several factors! This is what we were trying to address.

(PS If anyone does know a way to fix this, please make a pull request from staging and recommend what changes need to be made - your code will need to work across |RHEL / CentOS 5, 6, 7 and Ubuntu LTS Versions, and Debian)

  1. A workaround was therefore added with an arbitrary but informed limit of 50MB for the parent PID. Heres how I got to that figure:

Most parent PID's I have seen in apache are about 16 - 25 MB, its really only the children that bloat up, as they are the workers. so I took the 25MB and then doubled it to 50MB, which should give a reasonable level of growth allowance without being insane crazy big.

Normal apache parent PID sizes and child sizes:

Parent (16 - 25MB)
|-- Child (25 - 200MB+)
|-- Child (25 - 200MB+)
|-- Child (25 - 200MB+)

In this case, it is normal the children are using lots of memory, because they are working, and this is expected behaviour, the parent is quite small, and the children inherit a small footprint from the parent, but then grow in size as they are utilised (typically mod_php causing much of the bloat, but whatever, its normal.

Basically, if the parent PID is greater than 50MB, Apachebuddy will just abort, because the calculations are going to be so wild, it's not worth it.

However, I noticed that even with a bloated parent PID and subsequently children too, apache2buddy was still useful in those situations, as long as we went into it with our eyes open and prepared to gloss over the recommendations in the report.

In the early days I used to have to wget the file, edit it locally, and comment out the block that checked for the parent pid size. however, that quickly got tiresome, so I added in an option '--no-check-pid' which will do this on the fly.

So if, as you discover in your day to day operations, that apache2buddy wants to quit, because the parent pid is too big, its for good reason, the primary reason is:

apache2buddy really has one job, to calculate maxclients as accurately as possible, thats it, all the other stuff is just fluff, and you can check it out by looking at its grandfather 'apachebuddy.pl', that's it primary mission, if it doesnt feel it can deliver accurate results, it wants to get out of dodge city.

If you want the fluff, the extras, the cool stuff, and are willing to understand that the calculations for maxclients and memory usage might be way off, then you can do so by specifying

--no-check-pid

as a command line option.

with a curl and perl method, you need to specify an extra dash after the 'perl' otherwise linux thinks your asking it to call perl --no-check-pid:

curl -sL https://raw.githubusercontent.com/richardforth/apache2buddy/master/apache2buddy.pl | perl - --no-check-pid

Clone this wiki locally