How to measure Linux Performance Avoiding Most Typical Mistakes: Memory

Profile image of Peter Zaitsev

By Peter Zaitsev ✓ Guest post

July 06, 2020

This post is the third in a four-part blog series by Peter Zaitsev, Percona Chief Executive Officer.

In this series, we are talking about Linux performance measurement, how to measure it right. Linux performance is a very broad topic, so we’ll focus on the four primary resources which are typically going to drive your system performance – obviously CPU, memory, disk storage, and network.

Other parts in this series include:

In terms of memory, we look at the physical memory and virtual memory. Virtual memory is important because if we run out of virtual memory, we crash. Running out of Physical Memory causes slowness (due to paging).

Memory is classically important for driving performance. When you look at applications, especially like databases, you can think about memory both in terms of using the application processing as well as memory as a cache.

Memory as Cache

The cache is interesting because it essentially speeds up I/O performance. The cash, while it is memory, really impacts disk performance, not driven by memory performance.

Swap Used

One mistake which I see with virtual memory is your swap consumption like “Oh my gosh, I have something swapped out, that is a problem”.

It’s not a problem unless you are running out of swap or running out of virtual memory: then your system can crash.

The swap activity is what you should be watching. In this case let’s say that some minimal swap activity, maybe two kilobytes a second – not a big deal.

Swap activity on disk in Linux

Graph from Percona Monitoring and Management (PMM)

Looking at Virtual Memory Size

Another interesting fact: if you want to look at the memory usage by the applications, in many cases we look at the virtual memory and resident memory.

The problem with virtual memory is, especially in something like a Go application, it can be insane. Virtual memory size may exceed the amount of physical memory available many times and it is ok.

The problem for resident memory is that it doesn’t include the information that was swapped out. That can be problematic if a lot of processes are swapped out or it can be a process that you don’t expect to.

top on Linux showing virtual memory

Virtual memory size may exceed the amount of physical memory available many times and it is OK.

True Memory Usage by Process

The good thing is that both in the swap and in the processor in general there is actually a VM swap field which shows you how much of a process was swapped out, and that can help you to understand residence elapsed swap size.

Residence set size (RSS) shows how much of memory that process is actually using.

top on Linux showing SWAP size

Editorial note: to clarify, RSS means:

In computing, resident set size (RSS) is the portion of memory occupied by a process that is held in main memory (RAM). The rest of the occupied memory exists in the swap space or file system, either because some parts of the occupied memory were paged out, or because some parts of the executable were never loaded.[1]

Free Memory vs Available Memory

Some people are very concerned when you have free memory getting close to zero. But free memory getting close to zero is cool, that actually means, your kernel is using memory for a good purpose such as a cache.

Instead, you should look at the column of available memory - the memory which the kernel sees as available.

It can be used easily to throw away some pages. If your available memory is getting very low - that is when you would have some memory-driven problems and shortages.

Profile image of Peter Zaitsev

About Peter Zaitsev ✓ Guest post

Peter Zaitsev is CEO and co-founder of Percona. As one of the foremost experts on MySQL strategy and optimization, Peter leveraged both his technical vision and entrepreneurial skills. Peter is a co-author of High Performance MySQL: Optimization, Backups, and Replication, one of the most popular books on MySQL performance. Peter frequently speaks as an expert lecturer at conferences.

Want to subscribe to the cron.weekly newsletter?

I write a weekly-ish newsletter on Linux, open source & webdevelopment called cron.weekly.

It features the latest news, guides & tutorials and new open source projects. You can sign up via email below.

No spam. Just some good, practical Linux & open source content.