This is a hands-on journey to introduce cgroups are show how they work. You can follow along in a VM by running the commands in order. I will also be asking some questions for you to dig deeper and learn more about the topic. The questions are answered in this repository, which is a longer version of this post, or you can also find them in the man pages for cgroups v2.
What are cgroups?
cgroups control the amount of resources that processes are entitled to. In the context of containers they are very important to ensure that no single container hogs all of the resources of a system, leaving nothing for others.
cgroups v1
I am not covering cgroups v1 in this guide. To determine what cgroups version your system is using run the following command:
stat -fc %T /sys/fs/cgroup/The output should be:
cgroup2fsIf you want to know the differences between v1 and v2 read here or watch this talk. Or watch here for a list of issues with cgroup v1 that cgroup v2 solves.
1. Exploring the cgroups v2 filesystem
You can find the root of the cgroups filesystem under /sys/fs/cgroup:
ls /sys/fs/cgroup/
The directories here represent child cgroups. The ones present now,
init.scope,user.sliceandsystem.sliceare part of the system and I won't talk about them here but if you are curious you can learn more here.
All of your processes fall under the hierarchy of these directories and subdirectories, each directory representing a cgroup. The configuration of your controllers for each cgroup will determine how much resources are allocated to your processes.
There are two types of files you can read or read and write to:
Controller interface files (anything that doesn’t start with
cgroup.). For examplememory.maxto control the maximum amount of memory a cgroup can use.Core interface files starting with
cgroup.For examplecgroup.procsto assign processes to the cgroup.
Questions
What are the resource distribution models of these controllers?
What’s the difference between min, max and low?
What files are read-only and which ones are read and write?
2. Creating our cgroups hierarchy
Before we move on. Don’t run these commands on a Linux machine you directly work on or a production system, while none of them is likely to break your system, we are meddling with kernel settings and privileged commands here, so use a VM. I have used OrbStack on Mac for my VMs which I recommend.
Most of the commands that follow require privilege access, so let’s start with:
sudo suNow export these variables, feel free to change the values as they will represent your cgroup and child cgroup.
export ROOT_CGROUP=containers
export CONTAINER_CGROUP=goofytimesYou may need to install these:
apt -y install cgroup-toolsNow let’s create our parent cgroup
cgcreate -g memory,cpu:/${ROOT_CGROUP}You should get no out if successful, let’s look inside:
ls /sys/fs/cgroup/${PARENT_CGROUP}
Now let’s create the parent cgroup:
cgcreate -g memory,cpu:/${PARENT_CGROUP}/${CHILD_CGROUP}Now let’s check our child cgroup:
ls /sys/fs/cgroup/${PARENT_CGROUP}/${CHILD_CGROUP}
Question
The first cgcreate command was redundant as you could have just run the second one to create both cgroups, but I did it this way for a reason. When we created this cgroup, we specifically enabled only memory and cpu, why is pids also enabled here?
3. Adding controller configuration to our cgroup
We are going to set limits on the child cgroup to throttle the cpu and also induce an OOM kill.
Now we are going to to memory and cpu limits:
cgset -r memory.max=100000000 ${PARENT_CGROUP}/${CHILD_CGROUP}
cgset -r memory.swap.max=100000000 ${PARENT_CGROUP}/${CHILD_CGROUP}
cgset -r cpu.max="100000 1000000" ${PARENT_CGROUP}/${CHILD_CGROUP}Let’s check they have been set correctly:
cat /sys/fs/cgroup/${PARENT_CGROUP}/${CHILD_CGROUP}/{memory,cpu,memory.swap}.maxLet’s check our output:
Questions
Why are we setting both swap and memory limit to induce OOM kill?
What’s the difference between high and max limits? How does memory throttling compare?
Why did our byte input of
100000000turned to99999744Can we use different units other than bytes for memory?
What do the CPU numbers mean?
4. Testing the resource constrains our cgroup
Stressing the CPU
Let's start first by seeing what happens if we stress the CPU without a process being part of our cgroup:
yes >/dev/null &
sleep 0.5
ps -p $! -o %cpu
kill $!Output:
Now let's enter a bash terminal that's inside the cgroup we created like so:
cgexec -g memory,cpu:${PARENT_CGROUP}/${CHILD_CGROUP} bashAs you can see it only went up to 17.1%, but generally it will stay at around 10%
Filling up the memory
Let's now run this command within our bash cgrouped process from above:
echo $(tr -d '\0' < /dev/urandom | head -c200M) > /dev/null &This creates a very large variable of 200 MBs (which translates to more memory than this), which overloads the roughly 100 MB limit we imposed earlier. Hence, it will eventually kill the process, but in the meantime we can watch the memory and swap grow on the way to death with the following command:
watch ps -p $! -o rss,szOutput:
And eventually:
You can now exit your cgrouped shell:
exitQuestions
Set the
memory.maxandmemory.swap.maxtomaxand then set thememory.highandmemory.swap.highto the value100000000and observe the behaviour now. Does the OOM get triggered? What happens to the process instead?What strategies does the kernel use to try and throttle the memory? What’s the difference between high and max?
What happens if you try to run the following?:
cgexec -g memory,cpu:${PARENT_CGROUP} bashWhy does that fail?
5. cgroups top (table of processes) and ls (list)
systemd-cgls and systemd-cgtop are equivalent to ls and top for cgroups.
systemd-cgtop
First, it's worth noting that if a cgroup doesn't have any processes assigned to it, it will not show on systemd-cgtop, therefore we will run a few commands inside our cgroup to check it out:
for p in {1..5} ; do cgexec -g memory,cpu:${PARENT_CGROUP}/${CHILD_CGROUP} sleep 2000 & done
cgexec -g memory,cpu:${PARENT_CGROUP}/${CHILD_CGROUP} yes > /dev/null &Now that we have a few processes running let’s run this:
systemd-cgtop
systemd-cgls
Let’s check all our cgroups and their processes:
systemd-cgls /containers
6. Killing all processes in a cgroup
This one is fun, now that we are done with our processes, let's kill them all by running the following command:
echo 1 > /sys/fs/cgroup/${PARENT_CGROUP}/${CHILD_CGROUP}/cgroup.killAfter you press enter twice you will see all the processes have been killed:
7. Getting configuration parameters for a given cgroup
If you want to get an overview of the configuration parameters for a cgroup, we can use the command cgget
cgget ${PARENT_CGROUP}/${CHILD_CGROUP}Yields something like this (output truncated):
8. Cleaning up
You have reached the end of our journey and can go ahead and delete the cgroups we just created, this command will recursively delete both:
cgdelete -r -g cpu:/${PARENT_CGROUP}Question
Did you notice how in the commands cgexec and cgdelete we are specifying the controller even though it seemingly doesn't matter? Why do you think that is?
Resources
This is a list of resources that were helpful in the creation of this guide:
https://docs.kernel.org/admin-guide/cgroup-v2.html
https://medium.com/@charles.vissol/cgroup-introduction-45017140493d
https://medium.com/@charles.vissol/cgroup-v2-in-details-8c138088f9ba
https://medium.com/@charles.vissol/systemd-and-cgroup-7eb80a08234d
https://facebookmicrosites.github.io/cgroup2/docs/overview.html
https://kubernetes.io/docs/concepts/architecture/cgroups/
https://chrisdown.name/talks/cgroupv2/cgroupv2-fosdem.pdf
https://systemd.io/CGROUP_DELEGATION/
https://btholt.github.io/complete-intro-to-containers/
About the author
Fernando Villalba has over a decade of miscellaneous IT experience. He started in IT support ("Have you tried turning it on and off?"), veered to become a SysAdmin ("Don't you dare turn it off") and later segued into DevOps type of roles ("Destroy and replace!"). He has been a consultant for various multi-billion dollar organizations helping them achieve their highest potential with their DevOps processes.