Posted inlinux

Linux Cgroups v2 Memory Limits Tutorial

No Comments

Linux Cgroups v2 Memory Limits Tutorial

TL;DR: This linux cgroups v2 memory limits tutorial walks advanced system administrators through the unified cgroups v2 interface for memory control. You’ll learn how to enable cgroups v2, configure memory constraints, assign processes, and monitor usage—using real, root-level commands. Mastering cgroups v2 helps you prevent resource starvation, avoid OOM kills, and enforce fair memory allocation across containers and services.

Prerequisites

Before diving in, ensure you meet these requirements:

  • Linux Kernel 4.5 or newer (for full cgroups v2 support; 5.x+ recommended)
  • Root or sudo privileges (all cgroup management requires elevated access)
  • Familiarity with basic Linux concepts: mounting filesystems, editing bootloader configs, and process management
  • If using systemd (most modern distros): Knowledge of unit files and systemctl
  • On multi-boot systems: Understanding of distro-specific boot configs (GRUB, systemd-boot, etc.)

TIP: This guide covers direct cgroups v2 management via the /sys/fs/cgroup interface. If you use container orchestrators (Docker, Kubernetes), consult their docs for integration details. ()

TL;DR Summary

Cgroups v2 is the contemporary Linux kernel framework for fine-grained, hierarchical resource control—especially memory—across both processes and containers. It supersedes cgroups v1 with a unified hierarchy, consistent file naming, and more predictable enforcement. To leverage cgroups v2 for memory limits, the workflow is:

  • Enable cgroups v2: Ensure your kernel boots with systemd.unified_cgroup_hierarchy=1 (or equivalent).
  • Mount cgroup2: The cgroup2 filesystem should be mounted at /sys/fs/cgroup.
  • Create cgroup subtrees: Make directories for each workload or service within /sys/fs/cgroup.
  • Set memory limits: Write desired byte values to memory.max (hard limit), memory.high (soft limit), or memory.swap.max (swap limit).
  • Assign processes: Write process PIDs to the cgroup.procs file for each group.
  • Monitor usage: Inspect memory.current and memory.events for real-time stats and OOM events.

Key commands include mount, findmnt, cat, systemctl, and direct echo operations to cgroup files. This tutorial provides step-by-step, distro-specific instructions, real error output, and critical caveats to avoid downtime or accidental OOM kills.

NOTE: For a deeper dive into Linux resource control and other cgroups v2 examples, see our Linux Security Hardening Practices Tutorial.

Introduction to Cgroups v2

Contemporary Linux resource management relies heavily on cgroups v2—a kernel feature explicitly designed to partition and cap system resources such as memory, CPU, and I/O bandwidth. Cgroups let you allocate, prioritize, deny, and monitor resources for groups of processes, making them essential for container runtimes, multi-tenant servers, and predictable workload management.

History and Evolution of Cgroups

Cgroups originated as a response to the need for robust resource partitioning in the Linux kernel. Here’s a brief timeline:

  • Cgroups v1 (2008, Linux 2.6.24): Introduced with separate hierarchies for each controller (e.g., memory, cpu, blkio). While functional, v1’s design led to fragmentation, inconsistent behaviors, and complex setups—especially when controllers interacted in unexpected ways.
  • Cgroups v2 (2016, Linux 4.5+): Merged controllers into a unified hierarchy, providing a consistent interface and semantics. The cgroups v2 model also improves nested group enforcement and makes it easier for tools (like systemd, Docker, and Kubernetes) to interact seamlessly.

Modern distributions have moved toward cgroups v2 as the default, but some legacy environments may still default to v1 for compatibility reasons.

TIP: If you’re migrating from cgroups v1, review the differences in hierarchy and controller semantics before making changes to production systems. ()

Key Features of Cgroups v2

The most notable features that set cgroups v2 apart:

  • Unified Hierarchy: All resource controllers (memory, CPU, I/O, etc.) are managed in a single tree rooted at /sys/fs/cgroup. This eliminates the confusion of multiple parallel hierarchies in v1.
  • Consistent Interface: File names, formats, and controller semantics are standardized. For example, resource limits always use explicit files like memory.max.
  • Improved Resource Control: Hard and soft limits (memory.max, memory.high) are reliably enforced, including for nested cgroups.
  • Better Integration: Systemd (since v232) and most modern container runtimes can natively manage cgroups v2, reducing manual intervention and race conditions.
  • Predictable OOM Handling: OOM (Out Of Memory) events are tracked in memory.events, making it easier to debug and automate responses.

NOTE: Some older tools or scripts may only support cgroups v1. Always verify compatibility before migrating production systems. ()

Setting Up Cgroups v2

Cgroups v2 requires explicit kernel and filesystem configuration. On many modern distributions, it’s enabled by default, but legacy setups or custom kernels may need manual steps.

Enabling Cgroups v2

Step 1: Verify Kernel Support

First, check if your kernel supports cgroup2:

$ cat /proc/filesystems | grep cgroup2
nodev   cgroup2

If you see cgroup2, your kernel supports it.

Step 2: Check If cgroup2 is Mounted

$ grep cgroup2 /proc/mounts
none /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0

If the above returns nothing, cgroup2 is not mounted (or is mounted elsewhere).

Step 3: Set Kernel Boot Parameter

To ensure cgroups v2 is the default, the kernel should boot with a unified hierarchy. How you do this depends on your distro:

  • RHEL/CentOS 8+ (uses grubby):

“bash $ grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=1" “

  • Debian/Ubuntu 20.04+ (edit /etc/default/grub):

“ini GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1" ` Then apply: `bash $ update-grub “

  • Arch Linux: Usually defaults to cgroups v2. Verify with /proc/cmdline.

Step 4: Reboot

Apply kernel parameter changes by rebooting:

$ reboot

WARNING: Editing kernel boot parameters can render a system unbootable if misconfigured. Always keep a known-good kernel entry as a fallback. (Linux Security Hardening Practices Tutorial)

Step 5: Confirm Unified Hierarchy

After reboot:

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-5.15.0 root=/dev/sda1 ro systemd.unified_cgroup_hierarchy=1

Look for systemd.unified_cgroup_hierarchy=1 in the output.

Configuring Cgroups v2

With cgroup2 mounted at /sys/fs/cgroup, you can now create groups and set limits.

Example 1: Mount cgroup2 manually (if needed)

$ mount -t cgroup2 none /sys/fs/cgroup

This mounts the cgroup2 filesystem at the standard location.

Example 2: Verify cgroup2 mount

$ findmnt -t cgroup2
TARGET            SOURCE FSTYPE   OPTIONS
/sys/fs/cgroup    none   cgroup2  rw,nosuid,nodev,noexec,relatime

Example 3: Create a cgroup for a workload

$ mkdir /sys/fs/cgroup/web01

Creates a new cgroup subtree for a web server workload.

Example 4: Set a 1GB hard memory limit

$ echo 1073741824 > /sys/fs/cgroup/web01/memory.max

This enforces an upper limit of 1GB (in bytes) for all processes in /sys/fs/cgroup/web01.

Example 5: Assign a running process (PID 1234) to the cgroup

$ echo 1234 > /sys/fs/cgroup/web01/cgroup.procs

Moves the process to the specified cgroup.

Example 6: Set unlimited swap

$ echo "max" > /sys/fs/cgroup/web01/memory.swap.max

Allows the cgroup to use unlimited swap.

Example 7: Remove memory limit (set to unlimited)

$ echo "max" > /sys/fs/cgroup/web01/memory.max

Disables the hard memory cap.

WARNING: Writing to cgroup files requires root. Setting limits too low can OOM-kill critical processes immediately.

TIP: On systemd-based systems, direct changes under /sys/fs/cgroup may be overwritten on service restart. Prefer using systemctl set-property for persistent limits.

Managing Memory Limits with Cgroups v2

This section covers the heart of the linux cgroups v2 memory limits tutorial: setting, adjusting, and monitoring memory constraints for real workloads.

Setting Memory Limits

Example 1: Set a hard memory limit for a batch job

$ mkdir /sys/fs/cgroup/batch01
$ echo 536870912 > /sys/fs/cgroup/batch01/memory.max

Limits all processes in batch01 to 512MB RAM.

Example 2: Set a soft memory limit (memory.high) to 256MB

$ echo 268435456 > /sys/fs/cgroup/batch01/memory.high

Allows processes to exceed 256MB temporarily, but the kernel will reclaim memory under pressure.

Example 3: Limit swap usage to 128MB

$ echo 134217728 > /sys/fs/cgroup/batch01/memory.swap.max

Caps swap usage, which can prevent swap thrashing and unpredictable latency.

Example 4: Remove swap limit (unlimited swap)

$ echo max > /sys/fs/cgroup/batch01/memory.swap.max

Lets the cgroup use as much swap as is available.

Example 5: Set memory limit for a container runtime cgroup

$ mkdir -p /sys/fs/cgroup/docker/123abc
$ echo 2147483648 > /sys/fs/cgroup/docker/123abc/memory.max

Applies a 2GB hard limit to a Docker container’s cgroup.

NOTE: All memory values must be specified in bytes (not MB or GB). Using “512M” or “2G” will fail.

Systemd Integration Example:

For services managed by systemd (preferred on most modern distros):

$ systemctl set-property nginx.service MemoryMax=512M

This persists across reboots and service restarts.

TIP: Use systemctl show -p MemoryCurrent to check memory usage for a running systemd service. (Linux Security Hardening Practices Tutorial)

Monitoring Memory Usage

Once memory limits are set, it’s vital to monitor actual usage and detect OOM events.

Example 1: Check current memory usage

$ cat /sys/fs/cgroup/web01/memory.current
104857600

Shows usage in bytes (e.g., 100MB used).

Example 2: Inspect memory events (e.g., OOM, high water marks)

$ cat /sys/fs/cgroup/web01/memory.events
low 0
high 2
max 1
oom 0
oom_kill 1

This output means:

  • high 2: The cgroup hit its soft limit twice.
  • max 1: The hard limit was reached once.
  • oom_kill 1: One process was killed by the OOM killer.

Example 3: Monitor memory usage in real time

$ watch -n 1 'cat /sys/fs/cgroup/web01/memory.current'

Refreshes memory usage every second.

Example 4: List all cgroup2 mounts

$ findmnt -t cgroup2
TARGET            SOURCE FSTYPE   OPTIONS
/sys/fs/cgroup    none   cgroup2  rw,nosuid,nodev,noexec,relatime

Example 5: Check memory usage for all top-level cgroups

$ for d in /sys/fs/cgroup/*; do echo "$d: $(cat $d/memory.current 2>/dev/null)"; done

Quickly audits all cgroups’ memory consumption.

TIP: For systemd-managed services, use systemctl status to see memory usage, or read /sys/fs/cgroup/system.slice/.service/memory.current.

Common Mistakes & Gotchas

  1. Units confusion:

All memory values must be in bytes. Using 512M or 2G will throw an invalid argument error.

  1. Systemd overwrites manual changes:

Direct edits to /sys/fs/cgroup may be lost when systemd reloads or restarts a service. Use systemctl set-property for persistent settings.

  1. Incorrect PID assignment:

Writing a non-existent or wrong PID to cgroup.procs silently fails or assigns the wrong process. Always double-check PIDs with ps or pgrep.

  1. Hierarchy misunderstandings:

Parent cgroups’ limits are inherited by all children. If the parent is overly restrictive, children can’t exceed it—even if their own limits are higher.

  1. Not monitoring OOM events:

Failing to check memory.events can leave you blind to memory pressure or silent OOM kills.

  1. Forgetting to reboot after kernel param changes:

Kernel command line updates only take effect after reboot.

  1. Editing the wrong GRUB config:

On some distros, you must edit /etc/default/grub and regenerate the config. Editing /boot/grub/grub.cfg directly is almost always overwritten.

  1. Assigning processes before setting limits:

If you add a process before setting memory.max, it may exceed intended usage before the limit is enforced.

WARNING: Setting memory limits too low—especially on system services—can instantly OOM-kill vital infrastructure. Always test in a disposable environment. ()

Security & Production Considerations

  • Isolation boundaries:

Cgroups provide resource isolation, but not full security. Combine with Linux namespaces, capabilities, and seccomp for robust container isolation.

  • Root-only operations:

Only root can modify cgroup files directly. Use systemd drop-ins or container runtimes for multi-user environments.

  • Resource starvation:

Overly aggressive memory limits lead to service outages, OOM kills, and even kernel instability. Always test new limits in staging before production.

  • Auditing and drift detection:

Periodically audit /sys/fs/cgroup and systemd unit settings for unauthorized or accidental changes.

  • Performance tuning:

Use memory.high to set soft limits, allowing workloads to burst above their target without immediate OOM risk.

  • Monitoring and alerting:

Integrate memory.events and memory.current into your monitoring stack (Prometheus, Nagios, etc.) to detect and respond to memory pressure.

  • Container runtimes:

Docker and Kubernetes can manage cgroups automatically, but may require enabling cgroup v2 support via configuration flags. Always consult the runtime’s docs before deploying in production.

TIP: For advanced Linux performance tuning, combine cgroups v2 with CPU and I/O controllers for holistic resource governance. ()

Further Reading

NOTE: For more on memory management in Linux and related best practices, check out our Understanding the Lsmod Command in Linux.

Conclusion

Mastering cgroups v2 is essential for modern Linux performance tuning, predictable container behavior, and safe multi-tenant server operations. This linux cgroups v2 memory limits tutorial covered the why and how: from kernel boot parameters to real-world memory limit enforcement and monitoring. Remember, always use bytes for limit values, monitor events for OOMs, and leverage systemd for persistent configuration. By adopting cgroups v2 best practices, you’ll ensure your workloads get just the right slice of memory—no more, no less—while minimizing risk and maximizing system stability.

TIP: Ready to take it further? Experiment with other cgroup controllers (CPU, I/O), and integrate cgroup monitoring into your observability stack for end-to-end resource governance. ()

Tags:

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *