9:05 a.m.
Hi,
I recently moved a httpd mirror from i386 to x86_64 hardware. All but
the memory capacity were scaled up (100MBit to 1GBit, Athlon to
Opteron etc.).
The old server is running FC1/i386 and the new one got
FC2/x86_64. Although both distros have httpd in the same upstream
version (2.0.50) the memory footage is quite different, and leads to
the striking of the oom-killer on the new server:
old (FC1/i386):
root 3329 0.0 0.2 26900 2408 ? S Aug29 0:01 /usr/sbin/httpd
apache 32746 0.0 0.5 28388 5984 ? S 06:01 0:03 \_ /usr/sbin/httpd
apache 6206 0.0 0.5 28320 5968 ? S 09:31 0:02 \_ /usr/sbin/httpd
apache 8185 0.0 0.5 28452 6080 ? S 10:09 0:04 \_ /usr/sbin/httpd
apache 10293 0.0 0.5 28452 6056 ? S 11:07 0:03 \_ /usr/sbin/httpd
apache 10378 0.0 0.5 28396 5984 ? S 11:09 0:03 \_ /usr/sbin/httpd
[...]
new (FC2/x86_64):
root 2177 0.0 1.5 189664 15556 ? S Aug30 0:00 /usr/sbin/httpd
apache 2273 0.0 1.7 193076 17984 ? S Aug30 0:03 \_ /usr/sbin/httpd
apache 2282 0.0 1.7 193052 17996 ? S Aug30 0:04 \_ /usr/sbin/httpd
apache 3125 0.0 1.7 193052 17992 ? S Aug30 0:03 \_ /usr/sbin/httpd
apache 3993 0.0 1.7 193052 18060 ? S 01:27 0:03 \_ /usr/sbin/httpd
apache 3994 0.0 1.7 193052 17976 ? S 01:27 0:02 \_ /usr/sbin/httpd
apache 7193 0.0 1.7 193052 17992 ? S 05:01 0:02 \_ /usr/sbin/httpd
Is this normal for i386 -> x86_64? Anyone running httpd on x86_64 to
compare numbers?
Thanks!
--
Axel.Thimm at ATrpms.net
3:23 p.m.
On Tue, Aug 31, 2004 at 04:05:50PM +0200, Axel Thimm wrote:
Hi,
I recently moved a httpd mirror from i386 to x86_64 hardware. All but
the memory capacity were scaled up (100MBit to 1GBit, Athlon to
Opteron etc.).
The old server is running FC1/i386 and the new one got
FC2/x86_64. Although both distros have httpd in the same upstream
version (2.0.50) the memory footage is quite different, and leads to
the striking of the oom-killer on the new server:
I see the same thing here, not looked into it before though.
It's interesting, if you look at the mappings of the httpd process on
x86_64, for each mmaped object there is an extra region mapped with
PROT_NONE, which you don't see on i686. I presume this is counted in
the VmSize calculation - it adds up to about 100Mb of address space on
the system I tested.
e.g.
2a9b033000 12K r-xp /usr/lib64/libpanel.so.5.3
2a9b036000 1012K ---p /usr/lib64/libpanel.so.5.3
2a9b133000 16K rw-p /usr/lib64/libpanel.so.5.3
vs
00d57000 12K r-xp /usr/lib/libpanel.so.5.3
00d5a000 4K rw-p /usr/lib/libpanel.so.5.3
is this libc behaviour by design? Jakub?
old (FC1/i386):
root 3329 0.0 0.2 26900 2408 ? S Aug29 0:01 /usr/sbin/httpd
apache 32746 0.0 0.5 28388 5984 ? S 06:01 0:03 \_ /usr/sbin/httpd
apache 6206 0.0 0.5 28320 5968 ? S 09:31 0:02 \_ /usr/sbin/httpd
apache 8185 0.0 0.5 28452 6080 ? S 10:09 0:04 \_ /usr/sbin/httpd
apache 10293 0.0 0.5 28452 6056 ? S 11:07 0:03 \_ /usr/sbin/httpd
apache 10378 0.0 0.5 28396 5984 ? S 11:09 0:03 \_ /usr/sbin/httpd
[...]
new (FC2/x86_64):
root 2177 0.0 1.5 189664 15556 ? S Aug30 0:00 /usr/sbin/httpd
apache 2273 0.0 1.7 193076 17984 ? S Aug30 0:03 \_ /usr/sbin/httpd
apache 2282 0.0 1.7 193052 17996 ? S Aug30 0:04 \_ /usr/sbin/httpd
apache 3125 0.0 1.7 193052 17992 ? S Aug30 0:03 \_ /usr/sbin/httpd
apache 3993 0.0 1.7 193052 18060 ? S 01:27 0:03 \_ /usr/sbin/httpd
apache 3994 0.0 1.7 193052 17976 ? S 01:27 0:02 \_ /usr/sbin/httpd
apache 7193 0.0 1.7 193052 17992 ? S 05:01 0:02 \_ /usr/sbin/httpd
Is this normal for i386 -> x86_64? Anyone running httpd on x86_64 to
compare numbers?
Thanks!
--
Axel.Thimm at ATrpms.net
--
fedora-list mailing list
fedora-list(a)redhat.com
To unsubscribe: http://www.redhat.com/mailman/listinfo/fedora-list
3:34 p.m.
On Thu, Sep 02, 2004 at 09:23:10PM +0100, Joe Orton wrote:
I see the same thing here, not looked into it before though.
It's interesting, if you look at the mappings of the httpd process on
x86_64, for each mmaped object there is an extra region mapped with
PROT_NONE, which you don't see on i686. I presume this is counted in
the VmSize calculation - it adds up to about 100Mb of address space on
the system I tested.
e.g.
2a9b033000 12K r-xp /usr/lib64/libpanel.so.5.3
2a9b036000 1012K ---p /usr/lib64/libpanel.so.5.3
2a9b133000 16K rw-p /usr/lib64/libpanel.so.5.3
vs
00d57000 12K r-xp /usr/lib/libpanel.so.5.3
00d5a000 4K rw-p /usr/lib/libpanel.so.5.3
is this libc behaviour by design? Jakub?
Well, not glibc, but binutils.
The thing is, x86-64 ELF has 1MB pagesize, while i386 ELF 4KB,
so the x86-64 binaries and shared libraries must be usable even when kernel
uses 1MB pagesize.
The gap in between RE and RW segment is there so that the library occupies
less memory (eats less 4KB pages).
If something counts in PROT_NONE mappings into the process size, it should
be fixed.
Jakub
5:20 a.m.
On Thu, Sep 02, 2004 at 04:34:20PM -0400, Jakub Jelinek wrote:
On Thu, Sep 02, 2004 at 09:23:10PM +0100, Joe Orton wrote:
> I see the same thing here, not looked into it before though.
>
> It's interesting, if you look at the mappings of the httpd process on
> x86_64, for each mmaped object there is an extra region mapped with
> PROT_NONE, which you don't see on i686. I presume this is counted in
> the VmSize calculation - it adds up to about 100Mb of address space on
> the system I tested.
>
> e.g.
>
> 2a9b033000 12K r-xp /usr/lib64/libpanel.so.5.3
> 2a9b036000 1012K ---p /usr/lib64/libpanel.so.5.3
> 2a9b133000 16K rw-p /usr/lib64/libpanel.so.5.3
>
> vs
>
> 00d57000 12K r-xp /usr/lib/libpanel.so.5.3
> 00d5a000 4K rw-p /usr/lib/libpanel.so.5.3
>
> is this libc behaviour by design? Jakub?
Well, not glibc, but binutils.
The thing is, x86-64 ELF has 1MB pagesize, while i386 ELF 4KB,
so the x86-64 binaries and shared libraries must be usable even when kernel
uses 1MB pagesize.
The gap in between RE and RW segment is there so that the library occupies
less memory (eats less 4KB pages).
If something counts in PROT_NONE mappings into the process size, it should
be fixed.
Such mappings are certainly counted in the VmSize reported by
/proc/pid/status, which looks like it (mm->total_vm) is used in the OOM
killer heuristics and also in the 'ps' VSZ output. I couldn't argue
otherwise since it is vm space which is being "used".
But if these mappings are not really consuming memory, then they are not
necessarily the cause of your problems, Axel. It does mean that if you
*do* run out of memory, then the httpd processes are more likely to get
OOM killed on x86_64 than on i386.
joe
5:40 a.m.
On Fri, Sep 03, 2004 at 11:20:43AM +0100, Joe Orton wrote:
On Thu, Sep 02, 2004 at 04:34:20PM -0400, Jakub Jelinek wrote:
> On Thu, Sep 02, 2004 at 09:23:10PM +0100, Joe Orton wrote:
> > I see the same thing here, not looked into it before though.
> >
> > It's interesting, if you look at the mappings of the httpd process on
> > x86_64, for each mmaped object there is an extra region mapped with
> > PROT_NONE, which you don't see on i686. I presume this is counted in
> > the VmSize calculation - it adds up to about 100Mb of address space on
> > the system I tested.
> >
> > e.g.
> >
> > 2a9b033000 12K r-xp /usr/lib64/libpanel.so.5.3
> > 2a9b036000 1012K ---p /usr/lib64/libpanel.so.5.3
> > 2a9b133000 16K rw-p /usr/lib64/libpanel.so.5.3
> >
> > vs
> >
> > 00d57000 12K r-xp /usr/lib/libpanel.so.5.3
> > 00d5a000 4K rw-p /usr/lib/libpanel.so.5.3
> >
> > is this libc behaviour by design? Jakub?
>
> Well, not glibc, but binutils.
> The thing is, x86-64 ELF has 1MB pagesize, while i386 ELF 4KB,
> so the x86-64 binaries and shared libraries must be usable even when kernel
> uses 1MB pagesize.
>
> The gap in between RE and RW segment is there so that the library occupies
> less memory (eats less 4KB pages).
>
> If something counts in PROT_NONE mappings into the process size, it should
> be fixed.
Such mappings are certainly counted in the VmSize reported by
/proc/pid/status, which looks like it (mm->total_vm) is used in the OOM
killer heuristics and also in the 'ps' VSZ output. I couldn't argue
otherwise since it is vm space which is being "used".
But if these mappings are not really consuming memory, then they are not
necessarily the cause of your problems, Axel. It does mean that if you
*do* run out of memory, then the httpd processes are more likely to get
OOM killed on x86_64 than on i386.
Well, I can only argue phenomenologically, the same httpd config under
FC1/i386/1GB could easily serve the default 150 MaxClients. Migrating
to FC2/x86_64/1GB (e.g. same amount of RAM, but i386->x86_64 and
FC1->FC2) requires me to reduce MaxClients to below 50 to keep the
machine from running OOM.
The VM numbers are 8 times as big, and perhaps this figure is
irrelevant as discussed above, but the RSS size is also 3 times higher
than on the old server, and that is probably hurting (20MB RSS times
150 MaxClients is already at 3GB, while previously it was at 6MB times
resulting to 1.2GB).
How can I debug the memory consumption on this box? Which figures are
the ones to look for and which ones do accumulate for the OOM killer?
--
Axel.Thimm at ATrpms.net
5:46 a.m.
On Fri, Sep 03, 2004 at 12:40:53PM +0200, Axel Thimm wrote:
How can I debug the memory consumption on this box? Which figures
are
the ones to look for and which ones do accumulate for the OOM killer?
IMHO best would be to install 32-bit and 64-bit httpd side by side,
configure it the same (with a different port number),
keep downloading the same page from it and try to grab
/proc/<pid>/maps
from both processes.
Jakub
8:05 p.m.
New subject: 2.6.8-1.521 memory leak? (was: [...] FC1/i386 vs FC2/x86_64: [...] memory consumption?)
On Fri, Sep 03, 2004 at 06:46:18AM -0400, Jakub Jelinek wrote:
On Fri, Sep 03, 2004 at 12:40:53PM +0200, Axel Thimm wrote:
> How can I debug the memory consumption on this box? Which figures are
> the ones to look for and which ones do accumulate for the OOM killer?
IMHO best would be to install 32-bit and 64-bit httpd side by side,
configure it the same (with a different port number),
keep downloading the same page from it and try to grab
/proc/<pid>/maps
from both processes.
It turns out that memory gets consumed and not returned back to the
system independent of httpd (the oom-killer just strikes there first).
On an FC2/x86_64 system (Tyan S2880 with one processor only) with
untained 2.6.8-1.521 on 1GB RAM simple compilations can eat up all the
memory. I trimmed down such a system up to basic networking to detect
which processes were locking the memory, and no userland processes are
holding the memory. But almost all memory is flagged as "used" (with
negligible size of buffers and cache).
Is this a kernel memory leak? Any other information I should collect?
(I still cannot judge whether the change from kernel 2.4 to 2.6 or the
architecture change i386 to x86_64 is responsible for this due to lack
of different combinations)
# free
total used free shared buffers cached
Mem: 1027016 1022600 4416 0 992 7288
-/+ buffers/cache: 1014320 12696
Swap: 2047992 4496 2043496
# vmstat -a
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free inact active si so bi bo in cs us sy id wa
0 0 4496 4352 4548 6556 1 1 399 80 1517 162 2 2 88 8
# cat /proc/meminfo
MemTotal: 1027016 kB
MemFree: 4352 kB
Buffers: 1008 kB
Cached: 7316 kB
SwapCached: 1148 kB
Active: 6528 kB
Inactive: 4536 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 1027016 kB
LowFree: 4352 kB
SwapTotal: 2047992 kB
SwapFree: 2043496 kB
Dirty: 236 kB
Writeback: 0 kB
Mapped: 5296 kB
Slab: 14388 kB
Committed_AS: 535496 kB
PageTables: 494900 kB
VmallocTotal: 536870911 kB
VmallocUsed: 1568 kB
VmallocChunk: 536869323 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
# ps uaxwwf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 3472 428 ? S Sep12 0:01 init [3]
root 2 0.0 0.0 0 0 ? SWN Sep12 0:00 [ksoftirqd/0]
root 3 0.0 0.0 0 0 ? SW< Sep12 0:00 [events/0]
root 4 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [khelper]
root 5 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [kacpid]
root 30 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [kblockd/0]
root 44 0.0 0.0 0 0 ? SW Sep12 0:00 \_ [pdflush]
root 45 0.0 0.0 0 0 ? SW Sep12 0:02 \_ [pdflush]
root 47 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [aio/0]
root 186 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [ata/0]
root 31 0.0 0.0 0 0 ? SW Sep12 0:00 [khubd]
root 46 0.0 0.0 0 0 ? SW Sep12 0:01 [kswapd0]
root 151 0.0 0.0 0 0 ? SW Sep12 0:00 [kseriod]
root 188 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_0]
root 189 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_1]
root 204 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 339 0.0 0.0 2336 216 ? S< Sep12 0:00 udevd
root 896 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 897 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 898 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 899 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 1637 0.0 0.0 0 0 ? SW< Sep12 0:00 [krfcommd]
root 1946 0.0 0.0 18104 748 ? S Sep12 0:00 /usr/sbin/sshd
root 5189 0.0 0.1 37540 1056 ? S 02:04 0:00 \_ sshd: root@pts/0
root 5195 0.0 0.0 45656 1020 pts/0 S 02:04 0:00 | \_ -bash
root 5255 0.0 0.1 104764 1892 pts/0 S 02:04 0:00 | \_ gkrellm
root 29075 0.0 0.0 44836 500 pts/0 S 02:38 0:00 | \_ sleep 10
root 6119 0.0 0.0 37284 1020 ? S 02:19 0:00 \_ sshd: root@pts/1
root 6133 0.0 0.1 45656 1120 pts/1 S 02:19 0:00 | \_ -bash
root 29079 0.0 0.0 44476 924 pts/1 S 02:38 0:00 | \_ /bin/sh
./memory.sh
root 29083 0.0 0.0 5228 784 pts/1 R 02:38 0:00 | \_ ps uaxwwf
root 6193 0.0 0.0 37284 1020 ? S 02:20 0:00 \_ sshd: root@pts/2
root 6212 0.0 0.1 45656 1136 pts/2 S 02:20 0:00 | \_ -bash
root 29077 0.0 0.1 35936 1932 ? S 02:38 0:00 \_ sshd: bin [priv]
sshd 29078 0.0 0.1 19448 1120 ? S 02:38 0:00 \_ sshd: bin [net]
root 2542 0.0 0.0 2344 272 tty1 S Sep12 0:00 /sbin/mingetty tty1
root 2543 0.0 0.0 2344 272 tty2 S Sep12 0:00 /sbin/mingetty tty2
root 2544 0.0 0.0 2344 272 tty3 S Sep12 0:00 /sbin/mingetty tty3
root 2545 0.0 0.0 2344 276 tty4 S Sep12 0:00 /sbin/mingetty tty4
root 2546 0.0 0.0 2344 276 tty5 S Sep12 0:00 /sbin/mingetty tty5
root 2547 0.0 0.0 2344 276 tty6 S Sep12 0:00 /sbin/mingetty tty6
--
Axel.Thimm at ATrpms.net
4:36 a.m.
New subject: 2.6.8-1.521 memory leak? (was: [...] FC1/i386 vs FC2/x86_64: [...] memory consumption?)
As a follow-up, this looks familiar to the following bug reports in
bugzilla:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=131251
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=131414
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=132180
The latter two may be leaking due to the SG_IO/bio_uncopy_user memory
leak:
http://kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8....
But the system in question has no active device driven by SCSI layers
(no SATA/SCSI/CD-ROM attached, only plain-old IDE), and the memory
leak occurs when trying to rebuild 2.6.8-1.521 kernels or kernel
modules.
On Mon, Sep 13, 2004 at 03:05:44AM +0200, Axel Thimm wrote:
On Fri, Sep 03, 2004 at 06:46:18AM -0400, Jakub Jelinek wrote:
> On Fri, Sep 03, 2004 at 12:40:53PM +0200, Axel Thimm wrote:
> > How can I debug the memory consumption on this box? Which figures are
> > the ones to look for and which ones do accumulate for the OOM killer?
>
> IMHO best would be to install 32-bit and 64-bit httpd side by side,
> configure it the same (with a different port number),
> keep downloading the same page from it and try to grab
> /proc/<pid>/maps
> from both processes.
It turns out that memory gets consumed and not returned back to the
system independent of httpd (the oom-killer just strikes there first).
On an FC2/x86_64 system (Tyan S2880 with one processor only) with
untained 2.6.8-1.521 on 1GB RAM simple compilations can eat up all the
memory. I trimmed down such a system up to basic networking to detect
which processes were locking the memory, and no userland processes are
holding the memory. But almost all memory is flagged as "used" (with
negligible size of buffers and cache).
Is this a kernel memory leak? Any other information I should collect?
(I still cannot judge whether the change from kernel 2.4 to 2.6 or the
architecture change i386 to x86_64 is responsible for this due to lack
of different combinations)
# free
total used free shared buffers cached
Mem: 1027016 1022600 4416 0 992 7288
-/+ buffers/cache: 1014320 12696
Swap: 2047992 4496 2043496
# vmstat -a
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free inact active si so bi bo in cs us sy id wa
0 0 4496 4352 4548 6556 1 1 399 80 1517 162 2 2 88 8
# cat /proc/meminfo
MemTotal: 1027016 kB
MemFree: 4352 kB
Buffers: 1008 kB
Cached: 7316 kB
SwapCached: 1148 kB
Active: 6528 kB
Inactive: 4536 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 1027016 kB
LowFree: 4352 kB
SwapTotal: 2047992 kB
SwapFree: 2043496 kB
Dirty: 236 kB
Writeback: 0 kB
Mapped: 5296 kB
Slab: 14388 kB
Committed_AS: 535496 kB
PageTables: 494900 kB
VmallocTotal: 536870911 kB
VmallocUsed: 1568 kB
VmallocChunk: 536869323 kB
HugePages_Total: 0
HugePages_Free: 0
Hugepagesize: 2048 kB
# ps uaxwwf
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 3472 428 ? S Sep12 0:01 init [3]
root 2 0.0 0.0 0 0 ? SWN Sep12 0:00 [ksoftirqd/0]
root 3 0.0 0.0 0 0 ? SW< Sep12 0:00 [events/0]
root 4 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [khelper]
root 5 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [kacpid]
root 30 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [kblockd/0]
root 44 0.0 0.0 0 0 ? SW Sep12 0:00 \_ [pdflush]
root 45 0.0 0.0 0 0 ? SW Sep12 0:02 \_ [pdflush]
root 47 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [aio/0]
root 186 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [ata/0]
root 31 0.0 0.0 0 0 ? SW Sep12 0:00 [khubd]
root 46 0.0 0.0 0 0 ? SW Sep12 0:01 [kswapd0]
root 151 0.0 0.0 0 0 ? SW Sep12 0:00 [kseriod]
root 188 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_0]
root 189 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_1]
root 204 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 339 0.0 0.0 2336 216 ? S< Sep12 0:00 udevd
root 896 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 897 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 898 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 899 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
root 1637 0.0 0.0 0 0 ? SW< Sep12 0:00 [krfcommd]
root 1946 0.0 0.0 18104 748 ? S Sep12 0:00 /usr/sbin/sshd
root 5189 0.0 0.1 37540 1056 ? S 02:04 0:00 \_ sshd: root@pts/0
root 5195 0.0 0.0 45656 1020 pts/0 S 02:04 0:00 | \_ -bash
root 5255 0.0 0.1 104764 1892 pts/0 S 02:04 0:00 | \_ gkrellm
root 29075 0.0 0.0 44836 500 pts/0 S 02:38 0:00 | \_ sleep 10
root 6119 0.0 0.0 37284 1020 ? S 02:19 0:00 \_ sshd: root@pts/1
root 6133 0.0 0.1 45656 1120 pts/1 S 02:19 0:00 | \_ -bash
root 29079 0.0 0.0 44476 924 pts/1 S 02:38 0:00 | \_ /bin/sh
./memory.sh
root 29083 0.0 0.0 5228 784 pts/1 R 02:38 0:00 | \_ ps uaxwwf
root 6193 0.0 0.0 37284 1020 ? S 02:20 0:00 \_ sshd: root@pts/2
root 6212 0.0 0.1 45656 1136 pts/2 S 02:20 0:00 | \_ -bash
root 29077 0.0 0.1 35936 1932 ? S 02:38 0:00 \_ sshd: bin [priv]
sshd 29078 0.0 0.1 19448 1120 ? S 02:38 0:00 \_ sshd: bin [net]
root 2542 0.0 0.0 2344 272 tty1 S Sep12 0:00 /sbin/mingetty tty1
root 2543 0.0 0.0 2344 272 tty2 S Sep12 0:00 /sbin/mingetty tty2
root 2544 0.0 0.0 2344 272 tty3 S Sep12 0:00 /sbin/mingetty tty3
root 2545 0.0 0.0 2344 276 tty4 S Sep12 0:00 /sbin/mingetty tty4
root 2546 0.0 0.0 2344 276 tty5 S Sep12 0:00 /sbin/mingetty tty5
root 2547 0.0 0.0 2344 276 tty6 S Sep12 0:00 /sbin/mingetty tty6
--
Axel.Thimm at ATrpms.net
5:32 a.m.
New subject: kernel memory leak on x86_64 in ia32 emulation mode (was: 2.6.8-1.521 memory leak?)
On Mon, Sep 13, 2004 at 11:36:28AM +0200, Axel Thimm wrote:
On Mon, Sep 13, 2004 at 03:05:44AM +0200, Axel Thimm wrote:
> On Fri, Sep 03, 2004 at 06:46:18AM -0400, Jakub Jelinek wrote:
> > On Fri, Sep 03, 2004 at 12:40:53PM +0200, Axel Thimm wrote:
> > > How can I debug the memory consumption on this box? Which figures are
> > > the ones to look for and which ones do accumulate for the OOM killer?
> >
> > IMHO best would be to install 32-bit and 64-bit httpd side by side,
>
> It turns out that memory gets consumed and not returned back to the
> system independent of httpd (the oom-killer just strikes there first).
The bug has now been identified as a kernel memory leak in
2.6.8-1.521 and 2.6.8-1.541 on a x86_64 system running ia32 bit apps
in IA32_EMULATION mode.
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=132947
It is easily reproducable, and I wonder why there have been so few
reports on the net on this (kernel 2.6.8-1.521 has been released
before a month). Probably most x86_64 users are running in 64 bits
only and don't observe the leak.
Any references/fixes are more than welcome. :)
> On an FC2/x86_64 system (Tyan S2880 with one processor only)
with
> untained 2.6.8-1.521 on 1GB RAM simple compilations can eat up all the
> memory. I trimmed down such a system up to basic networking to detect
> which processes were locking the memory, and no userland processes are
> holding the memory. But almost all memory is flagged as "used" (with
> negligible size of buffers and cache).
>
> Is this a kernel memory leak? Any other information I should collect?
>
> (I still cannot judge whether the change from kernel 2.4 to 2.6 or the
> architecture change i386 to x86_64 is responsible for this due to lack
> of different combinations)
>
> # free
> total used free shared buffers cached
> Mem: 1027016 1022600 4416 0 992 7288
> -/+ buffers/cache: 1014320 12696
> Swap: 2047992 4496 2043496
> # vmstat -a
> procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
> r b swpd free inact active si so bi bo in cs us sy id wa
> 0 0 4496 4352 4548 6556 1 1 399 80 1517 162 2 2 88 8
> # cat /proc/meminfo
> MemTotal: 1027016 kB
> MemFree: 4352 kB
> Buffers: 1008 kB
> Cached: 7316 kB
> SwapCached: 1148 kB
> Active: 6528 kB
> Inactive: 4536 kB
> HighTotal: 0 kB
> HighFree: 0 kB
> LowTotal: 1027016 kB
> LowFree: 4352 kB
> SwapTotal: 2047992 kB
> SwapFree: 2043496 kB
> Dirty: 236 kB
> Writeback: 0 kB
> Mapped: 5296 kB
> Slab: 14388 kB
> Committed_AS: 535496 kB
> PageTables: 494900 kB
> VmallocTotal: 536870911 kB
> VmallocUsed: 1568 kB
> VmallocChunk: 536869323 kB
> HugePages_Total: 0
> HugePages_Free: 0
> Hugepagesize: 2048 kB
> # ps uaxwwf
> USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
> root 1 0.0 0.0 3472 428 ? S Sep12 0:01 init [3]
> root 2 0.0 0.0 0 0 ? SWN Sep12 0:00 [ksoftirqd/0]
> root 3 0.0 0.0 0 0 ? SW< Sep12 0:00 [events/0]
> root 4 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [khelper]
> root 5 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [kacpid]
> root 30 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [kblockd/0]
> root 44 0.0 0.0 0 0 ? SW Sep12 0:00 \_ [pdflush]
> root 45 0.0 0.0 0 0 ? SW Sep12 0:02 \_ [pdflush]
> root 47 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [aio/0]
> root 186 0.0 0.0 0 0 ? SW< Sep12 0:00 \_ [ata/0]
> root 31 0.0 0.0 0 0 ? SW Sep12 0:00 [khubd]
> root 46 0.0 0.0 0 0 ? SW Sep12 0:01 [kswapd0]
> root 151 0.0 0.0 0 0 ? SW Sep12 0:00 [kseriod]
> root 188 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_0]
> root 189 0.0 0.0 0 0 ? SW Sep12 0:00 [scsi_eh_1]
> root 204 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
> root 339 0.0 0.0 2336 216 ? S< Sep12 0:00 udevd
> root 896 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
> root 897 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
> root 898 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
> root 899 0.0 0.0 0 0 ? SW Sep12 0:00 [kjournald]
> root 1637 0.0 0.0 0 0 ? SW< Sep12 0:00 [krfcommd]
> root 1946 0.0 0.0 18104 748 ? S Sep12 0:00 /usr/sbin/sshd
> root 5189 0.0 0.1 37540 1056 ? S 02:04 0:00 \_ sshd: root@pts/0
> root 5195 0.0 0.0 45656 1020 pts/0 S 02:04 0:00 | \_ -bash
> root 5255 0.0 0.1 104764 1892 pts/0 S 02:04 0:00 | \_ gkrellm
> root 29075 0.0 0.0 44836 500 pts/0 S 02:38 0:00 | \_ sleep 10
> root 6119 0.0 0.0 37284 1020 ? S 02:19 0:00 \_ sshd: root@pts/1
> root 6133 0.0 0.1 45656 1120 pts/1 S 02:19 0:00 | \_ -bash
> root 29079 0.0 0.0 44476 924 pts/1 S 02:38 0:00 | \_ /bin/sh
./memory.sh
> root 29083 0.0 0.0 5228 784 pts/1 R 02:38 0:00 | \_ ps
uaxwwf
> root 6193 0.0 0.0 37284 1020 ? S 02:20 0:00 \_ sshd: root@pts/2
> root 6212 0.0 0.1 45656 1136 pts/2 S 02:20 0:00 | \_ -bash
> root 29077 0.0 0.1 35936 1932 ? S 02:38 0:00 \_ sshd: bin [priv]
> sshd 29078 0.0 0.1 19448 1120 ? S 02:38 0:00 \_ sshd: bin
[net]
> root 2542 0.0 0.0 2344 272 tty1 S Sep12 0:00 /sbin/mingetty tty1
> root 2543 0.0 0.0 2344 272 tty2 S Sep12 0:00 /sbin/mingetty tty2
> root 2544 0.0 0.0 2344 272 tty3 S Sep12 0:00 /sbin/mingetty tty3
> root 2545 0.0 0.0 2344 276 tty4 S Sep12 0:00 /sbin/mingetty tty4
> root 2546 0.0 0.0 2344 276 tty5 S Sep12 0:00 /sbin/mingetty tty5
> root 2547 0.0 0.0 2344 276 tty6 S Sep12 0:00 /sbin/mingetty tty6
--
Axel.Thimm at ATrpms.net
6:32 a.m.
Actually, I don't see RSS numbers particularly different between i686
and x86_64.
For a server under ab load for a static page, I get an RSS of ~10.5Mb on
amd64, and ~9.5Mb on i686, which seems like a reasonable difference.
This is comparing like for like on RHEL3 U3, I don't have a Hammer box
running FC2 to compare.
You are comparing active servers not idle ones, right? RSS of an idle
server is not interesting since it may be half swapped out.
joe
7:54 a.m.
On Fri, Sep 03, 2004 at 12:32:42PM +0100, Joe Orton wrote:
Actually, I don't see RSS numbers particularly different between
i686
and x86_64.
For a server under ab load for a static page, I get an RSS of ~10.5Mb on
amd64, and ~9.5Mb on i686, which seems like a reasonable difference.
This is comparing like for like on RHEL3 U3, I don't have a Hammer box
running FC2 to compare.
In my case this was 20MB vs 6MB.
You are comparing active servers not idle ones, right? RSS of an
idle
server is not interesting since it may be half swapped out.
No, the numbers were from actual full workload scenarios.
It must not be a i386 vs x86_64 issue, it can also be FC1->FC2 issue
(including kernel 2.4->2.6).
--
Axel.Thimm at ATrpms.net
11:16 a.m.
On Fri, Sep 03, 2004 at 02:54:12PM +0200, Axel Thimm wrote:
It must not be a i386 vs x86_64 issue, it can also be FC1->FC2
issue
(including kernel 2.4->2.6).
I see roughly the same RSS and VSZ between FC1 and FC2 on x86 boxes. I
presume you have exactly the same set of modules installed&loaded on
both servers?
joe
11:21 a.m.
On Fri, Sep 03, 2004 at 05:16:51PM +0100, Joe Orton wrote:
On Fri, Sep 03, 2004 at 02:54:12PM +0200, Axel Thimm wrote:
> It must not be a i386 vs x86_64 issue, it can also be FC1->FC2 issue
> (including kernel 2.4->2.6).
I see roughly the same RSS and VSZ between FC1 and FC2 on x86 boxes. I
presume you have exactly the same set of modules installed&loaded on
both servers?
I am using the default set of modules that FC1 resp. FC2 come
preconfigured with, I only configure vhosts upon the default config.
Is there such a difference FC1 vs FC2 in the set of default modules
and the thus consumed memory?
--
Axel.Thimm at ATrpms.net
7193
days inactive
7213
days old
12 comments
3 participants
participants (3)
-
Axel Thimm -
Jakub Jelinek -
Joe Orton