This post is the third in a four-part blog series by Peter Zaitsev, Percona Chief Executive Officer.

In this series, we are talking about Linux performance measurement, how to measure it right. Linux performance is a very broad topic, so we’ll focus on the four primary resources which are typically going to drive your system performance – obviously CPU, memory, disk storage, and network.

Other parts in this series include:

• Measuring Linux Performance: CPU
• Measuring Linux Performance: Disk
• Measuring Linux Performance: Memory
• Measuring Linux Performance: Network

In terms of memory, we look at the physical memory and virtual memory. Virtual memory is important because if we run out of virtual memory, we crash. Running out of Physical Memory causes slowness (due to paging).

Memory is classically important for driving performance. When you look at applications, especially like databases, you can think about memory both in terms of using the application processing as well as memory as a cache.

Memory as Cache

The cache is interesting because it essentially speeds up I/O performance. The cash, while it is memory, really impacts disk performance, not driven by memory performance.

Swap Used

One mistake which I see with virtual memory is your swap consumption like “Oh my gosh, I have something swapped out, that is a problem”.

It’s not a problem unless you are running out of swap or running out of virtual memory: then your system can crash.

The swap activity is what you should be watching. In this case let’s say that some minimal swap activity, maybe two kilobytes a second – not a big deal.

Graph from Percona Monitoring and Management (PMM)

Looking at Virtual Memory Size

Another interesting fact: if you want to look at the memory usage by the applications, in many cases we look at the virtual memory and resident memory.

The problem with virtual memory is, especially in something like a Go application, it can be insane. Virtual memory size may exceed the amount of physical memory available many times and it is ok.

The problem for resident memory is that it doesn’t include the information that was swapped out. That can be problematic if a lot of processes are swapped out or it can be a process that you don’t expect to.

Virtual memory size may exceed the amount of physical memory available many times and it is OK.

True Memory Usage by Process

The good thing is that both in the swap and in the processor in general there is actually a VM swap field which shows you how much of a process was swapped out, and that can help you to understand residence elapsed swap size.

Residence set size (RSS) shows how much of memory that process is actually using.

Editorial note: to clarify, RSS means:

In computing, resident set size (RSS) is the portion of memory occupied by a process that is held in main memory (RAM). The rest of the occupied memory exists in the swap space or file system, either because some parts of the occupied memory were paged out, or because some parts of the executable were never loaded.[1]

Free Memory vs Available Memory

Some people are very concerned when you have free memory getting close to zero. But free memory getting close to zero is cool, that actually means, your kernel is using memory for a good purpose such as a cache.

Instead, you should look at the column of available memory - the memory which the kernel sees as available.

It can be used easily to throw away some pages. If your available memory is getting very low - that is when you would have some memory-driven problems and shortages.