1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
|
The Resource Counter
The resource counter, declared at include/linux/res_counter.h,
is supposed to facilitate the resource management by controllers
by providing common stuff for accounting.
This "stuff" includes the res_counter structure and routines
to work with it.
1. Crucial parts of the res_counter structure
a. unsigned long long usage
The usage value shows the amount of a resource that is consumed
by a group at a given time. The units of measurement should be
determined by the controller that uses this counter. E.g. it can
be bytes, items or any other unit the controller operates on.
b. unsigned long long max_usage
The maximal value of the usage over time.
This value is useful when gathering statistical information about
the particular group, as it shows the actual resource requirements
for a particular group, not just some usage snapshot.
c. unsigned long long limit
The maximal allowed amount of resource to consume by the group. In
case the group requests for more resources, so that the usage value
would exceed the limit, the resource allocation is rejected (see
the next section).
d. unsigned long long failcnt
The failcnt stands for "failures counter". This is the number of
resource allocation attempts that failed.
c. spinlock_t lock
Protects changes of the above values.
2. Basic accounting routines
a. void res_counter_init(struct res_counter *rc,
struct res_counter *rc_parent)
Initializes the resource counter. As usual, should be the first
routine called for a new counter.
The struct res_counter *parent can be used to define a hierarchical
child -> parent relationship directly in the res_counter structure,
NULL can be used to define no relationship.
c. int res_counter_charge(struct res_counter *rc, unsigned long val,
struct res_counter **limit_fail_at)
When a resource is about to be allocated it has to be accounted
with the appropriate resource counter (controller should determine
which one to use on its own). This operation is called "charging".
This is not very important which operation - resource allocation
or charging - is performed first, but
* if the allocation is performed first, this may create a
temporary resource over-usage by the time resource counter is
charged;
* if the charging is performed first, then it should be uncharged
on error path (if the one is called).
If the charging fails and a hierarchical dependency exists, the
limit_fail_at parameter is set to the particular res_counter element
where the charging failed.
d. int res_counter_charge_locked
(struct res_counter *rc, unsigned long val, bool force)
The same as res_counter_charge(), but it must not acquire/release the
res_counter->lock internally (it must be called with res_counter->lock
held). The force parameter indicates whether we can bypass the limit.
e. void res_counter_uncharge[_locked]
(struct res_counter *rc, unsigned long val)
When a resource is released (freed) it should be de-accounted
from the resource counter it was accounted to. This is called
"uncharging".
The _locked routines imply that the res_counter->lock is taken.
f. void res_counter_uncharge_until
(struct res_counter *rc, struct res_counter *top,
unsinged long val)
Almost same as res_cunter_uncharge() but propagation of uncharge
stops when rc == top. This is useful when kill a res_coutner in
child cgroup.
2.1 Other accounting routines
There are more routines that may help you with common needs, like
checking whether the limit is reached or resetting the max_usage
value. They are all declared in include/linux/res_counter.h.
3. Analyzing the resource counter registrations
a. If the failcnt value constantly grows, this means that the counter's
limit is too tight. Either the group is misbehaving and consumes too
many resources, or the configuration is not suitable for the group
and the limit should be increased.
b. The max_usage value can be used to quickly tune the group. One may
set the limits to maximal values and either load the container with
a common pattern or leave one for a while. After this the max_usage
value shows the amount of memory the container would require during
its common activity.
Setting the limit a bit above this value gives a pretty good
configuration that works in most of the cases.
c. If the max_usage is much less than the limit, but the failcnt value
is growing, then the group tries to allocate a big chunk of resource
at once.
d. If the max_usage is much less than the limit, but the failcnt value
is 0, then this group is given too high limit, that it does not
require. It is better to lower the limit a bit leaving more resource
for other groups.
4. Communication with the control groups subsystem (cgroups)
All the resource controllers that are using cgroups and resource counters
should provide files (in the cgroup filesystem) to work with the resource
counter fields. They are recommended to adhere to the following rules:
a. File names
Field name File name
---------------------------------------------------
usage usage_in_<unit_of_measurement>
max_usage max_usage_in_<unit_of_measurement>
limit limit_in_<unit_of_measurement>
failcnt failcnt
lock no file :)
b. Reading from file should show the corresponding field value in the
appropriate format.
c. Writing to file
Field Expected behavior
----------------------------------
usage prohibited
max_usage reset to usage
limit set the limit
failcnt reset to zero
5. Usage example
a. Declare a task group (take a look at cgroups subsystem for this) and
fold a res_counter into it
struct my_group {
struct res_counter res;
<other fields>
}
b. Put hooks in resource allocation/release paths
int alloc_something(...)
{
if (res_counter_charge(res_counter_ptr, amount) < 0)
return -ENOMEM;
<allocate the resource and return to the caller>
}
void release_something(...)
{
res_counter_uncharge(res_counter_ptr, amount);
<release the resource>
}
In order to keep the usage value self-consistent, both the
"res_counter_ptr" and the "amount" in release_something() should be
the same as they were in the alloc_something() when the releasing
resource was allocated.
c. Provide the way to read res_counter values and set them (the cgroups
still can help with it).
c. Compile and run :)
|
