summaryrefslogtreecommitdiffstats
path: root/include/linux/rcupdate.h
diff options
context:
space:
mode:
authorBoqun Feng <boqun.feng@gmail.com>2015-07-30 16:55:38 -0700
committerPaul E. McKenney <paulmck@linux.vnet.ibm.com>2015-10-06 11:08:23 -0700
commitbb73c52bad3666997ed2ec83c0c80c3f8ef55008 (patch)
tree5dd6150075147c23984f6c4902ba5fcde4612338 /include/linux/rcupdate.h
parentdb3e8db45e1cbf8cc22f1083a559d78eb91ccd63 (diff)
downloadblackbird-op-linux-bb73c52bad3666997ed2ec83c0c80c3f8ef55008.tar.gz
blackbird-op-linux-bb73c52bad3666997ed2ec83c0c80c3f8ef55008.zip
rcu: Don't disable preemption for Tiny and Tree RCU readers
Because preempt_disable() maps to barrier() for non-debug builds, it forces the compiler to spill and reload registers. Because Tree RCU and Tiny RCU now only appear in CONFIG_PREEMPT=n builds, these barrier() instances generate needless extra code for each instance of rcu_read_lock() and rcu_read_unlock(). This extra code slows down Tree RCU and bloats Tiny RCU. This commit therefore removes the preempt_disable() and preempt_enable() from the non-preemptible implementations of __rcu_read_lock() and __rcu_read_unlock(), respectively. However, for debug purposes, preempt_disable() and preempt_enable() are still invoked if CONFIG_PREEMPT_COUNT=y, because this allows detection of sleeping inside atomic sections in non-preemptible kernels. However, Tiny and Tree RCU operates by coalescing all RCU read-side critical sections on a given CPU that lie between successive quiescent states. It is therefore necessary to compensate for removing barriers from __rcu_read_lock() and __rcu_read_unlock() by adding them to a couple of the RCU functions invoked during quiescent states, namely to rcu_all_qs() and rcu_note_context_switch(). However, note that the latter is more paranoia than necessity, at least until link-time optimizations become more aggressive. This is based on an earlier patch by Paul E. McKenney, fixing a bug encountered in kernels built with CONFIG_PREEMPT=n and CONFIG_PREEMPT_COUNT=y. Signed-off-by: Boqun Feng <boqun.feng@gmail.com> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Diffstat (limited to 'include/linux/rcupdate.h')
-rw-r--r--include/linux/rcupdate.h6
1 files changed, 4 insertions, 2 deletions
diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
index d63bb77dab35..6c3ceceb6148 100644
--- a/include/linux/rcupdate.h
+++ b/include/linux/rcupdate.h
@@ -297,12 +297,14 @@ void synchronize_rcu(void);
static inline void __rcu_read_lock(void)
{
- preempt_disable();
+ if (IS_ENABLED(CONFIG_PREEMPT_COUNT))
+ preempt_disable();
}
static inline void __rcu_read_unlock(void)
{
- preempt_enable();
+ if (IS_ENABLED(CONFIG_PREEMPT_COUNT))
+ preempt_enable();
}
static inline void synchronize_rcu(void)
OpenPOWER on IntegriCloud