talos-op-linux/arch/powerpc/kernel/vdso64, branch v4.4.4

powerpc/vdso: Avoid link stack corruption in __get_datapage()

2015-10-01T06:52:02+00:00

powerpc has a link register (lr) used for calling functions. We "bl " to call a function, and "blr" to return back to the call site. The lr is only a single register, so if we call another function from inside this function (ie. nested calls), software must save away the lr on the software stack before calling the new function. Before returning (ie. before the "blr"), the lr is restored by software from the software stack. This makes branch prediction quite difficult for the processor as it will only know the branch target just before the "blr". To help with this, modern powerpc processors keep a (non-architected) hardware stack of lr called a "link stack". When a "bl " is run, the lr is pushed onto this stack. When a "blr" is called, the branch predictor pops the lr value from the top of the link stack, and uses it to predict the branch target. Hence the processor pipeline knows a lot earlier the branch target. This works great but there are some cases where you call "bl" but without a matching "blr". Once such case is when trying to determine the program counter (which can't be read directly). Here you "bl+4; mflr" to get the program counter. If you do this, the link stack will get out of sync with reality, causing the branch predictor to mis-predict subsequent function returns. To avoid this, modern micro-architectures have a special case of bl. Using the form "bcl 20,31,+4", ensures the processor doesn't push to the link stack. The 32 and 64 bit variants of __get_datapage() use a "bl; mflr" to determine the loaded address of the VDSO. The current versions of these attempt to use this special bl variant. Unfortunately they use +8 rather than the required +4. Hence the current code results in the link stack getting out of sync with reality and hence the resulting performance degradation. This patch moves it to bcl+4 by moving __kernel_datapage_offset out of __get_datapage(). With this patch, running a gettimeofday() (which uses __get_datapage()) microbenchmark we get a decent bump in performance on POWER7/8. For the benchmark in tools/testing/selftests/powerpc/benchmarks/gettimeofday.c POWER8: 64bit gets ~4% improvement 32bit gets ~9% improvement POWER7: 64bit gets ~7% improvement Signed-off-by: Michael Neuling Reported-by: Aaron Sawdey Signed-off-by: Michael Ellerman

powerpc/vdso: Emit GNU & SysV hashes

2015-10-01T06:52:00+00:00

Andy Lutomirski says: Some dynamic loaders may be slightly faster if a GNU hash is available. This is unlikely to have any measurable effect on the time it takes to resolve vdso symbols (since there are so few of them). In some contexts, it can be a win for a different reason: if every DSO has a GNU hash section, then libc can avoid calculating SysV hashes at all. Both musl and glibc appear to have this optimization. Signed-off-by: Michael Ellerman

powerpc/booke64: Use SPRG7 for VDSO

2014-03-20T00:57:14+00:00

Previously SPRG3 was marked for use by both VDSO and critical interrupts (though critical interrupts were not fully implemented). In commit 8b64a9dfb091f1eca8b7e58da82f1e7d1d5fe0ad ("powerpc/booke64: Use SPRG0/3 scratch for bolted TLB miss & crit int"), Mihai Caraman made an attempt to resolve this conflict by restoring the VDSO value early in the critical interrupt, but this has some issues: - It's incompatible with EXCEPTION_COMMON which restores r13 from the by-then-overwritten scratch (this cost me some debugging time). - It forces critical exceptions to be a special case handled differently from even machine check and debug level exceptions. - It didn't occur to me that it was possible to make this work at all (by doing a final "ld r13, PACA_EXCRIT+EX_R13(r13)") until after I made (most of) this patch. :-) It might be worth investigating using a load rather than SPRG on return from all exceptions (except TLB misses where the scratch never leaves the SPRG) -- it could save a few cycles. Until then, let's stick with SPRG for all exceptions. Since we cannot use SPRG4-7 for scratch without corrupting the state of a KVM guest, move VDSO to SPRG7 on book3e. Since neither SPRG4-7 nor critical interrupts exist on book3s, SPRG3 is still used for VDSO there. Signed-off-by: Scott Wood Cc: Mihai Caraman Cc: Anton Blanchard Cc: Paul Mackerras Cc: kvm-ppc@vger.kernel.org

powerpc: Use unstripped VDSO image for more accurate profiling data

2014-02-17T00:19:37+00:00

We are seeing a lot of hits in the VDSO that are not resolved by perf. A while(1) gettimeofday() loop shows the issue: 27.64% [vdso] [.] 0x000000000000060c 22.57% [vdso] [.] 0x0000000000000628 16.88% [vdso] [.] 0x0000000000000610 12.39% [vdso] [.] __kernel_gettimeofday 6.09% [vdso] [.] 0x00000000000005f8 3.58% test [.] 00000037.plt_call.gettimeofday@@GLIBC_2.18 2.94% [vdso] [.] __kernel_datapage_offset 2.90% test [.] main We are using a stripped VDSO image which means only symbols with relocation info can be resolved. There isn't a lot of point to stripping the VDSO, the debug info is only about 1kB: 4680 arch/powerpc/kernel/vdso64/vdso64.so 5815 arch/powerpc/kernel/vdso64/vdso64.so.dbg By using the unstripped image, we can resolve all the symbols in the VDSO and the perf profile data looks much better: 76.53% [vdso] [.] __do_get_tspec 12.20% [vdso] [.] __kernel_gettimeofday 5.05% [vdso] [.] __get_datapage 3.20% test [.] main 2.92% test [.] 00000037.plt_call.gettimeofday@@GLIBC_2.18 Signed-off-by: Anton Blanchard Signed-off-by: Benjamin Herrenschmidt

powerpc: Delete non-required instances of include

2014-01-15T02:46:44+00:00

None of these files are actually using any __init type directives and hence don't need to include . Most are just a left over from __devinit and __cpuinit removal, or simply due to code getting copied from one driver to the next. The one instance where we add an include for init.h covers off a case where that file was implicitly getting it from another header which itself didn't need it. Signed-off-by: Paul Gortmaker Signed-off-by: Benjamin Herrenschmidt

powerpc: Wrong DWARF CFI in the kernel vdso for little-endian / ELFv2

2013-11-20T22:19:23+00:00

I've finally tracked down why my CR signal-unwind test case still fails on little-endian. The problem turned to be that the kernel installs a signal trampoline in the vDSO, and provides a DWARF CFI record for that trampoline. This CFI describes the save location for CR: rsave (70, 38*RSIZE + (RSIZE - CRSIZE)) which is correct for big-endian, but points to the wrong word on little-endian. This is wrong no matter which ABI. In addition, for the ELFv2 ABI, we should not only provide a CFI record for register 70 (cr2), but for all CR fields separately. Strictly speaking, I guess this would mean providing two separate vDSO images, one for ELFv1 processes and one for ELFv2 processes (or maybe playing some tricks with conditional DWARF expressions). However, having CFI records for the other CR fields in ELFv1 is not actually wrong, they just will be ignored. So it seems the simplest fix would be just to always provide CFI for all the fields. Signed-off-by: Ulrich Weigand Signed-off-by: Anton Blanchard Signed-off-by: Benjamin Herrenschmidt

powerpc: Add ability to build little endian kernels

2013-10-11T05:48:56+00:00

This patch allows the kbuild system to successfully compile a kernel for the little endian PowerPC64 architecture. A subsequent patch will add the CONFIG_CPU_LITTLE_ENDIAN kernel config option which must be set to build such a kernel. If cross compiling, CROSS_COMPILE must point to a suitable toolchain (compiled for the powerpc64le-linux and powerpcle-linux targets). Signed-off-by: Ian Munsie Signed-off-by: Anton Blanchard Signed-off-by: Benjamin Herrenschmidt

powerpc: Add VDSO version of time

2013-04-23T06:05:05+00:00

On 04/18/2013 07:38 PM, Anton Blanchard wrote: > Since you are only reading one long you shouldn't need to check the > update count and loop, you will always see a consistent value. The > system call version of time() just does an unprotected load for example. Fixed. > With the above change and with Michael's comments covered (decent > changelog entry and Signed-off-by): > > Acked-by: Anton Blanchard Thanks for the review, below the updated patch: From: Adhemerval Zanella This patch implement the time syscall as vDSO. The performance speedups are: Baseline PPC32: 380 nsec Baseline PPC64: 350 nsec vdso PPC32: 20 nsec vsdo PPC64: 20 nsec Tested on 64 bit build with both 32 bit and 64 bit userland. Acked-by: Anton Blanchard Signed-off-by: Adhemerval Zanella Signed-off-by: Benjamin Herrenschmidt

powerpc: Add VDSO version of getcpu

2012-07-11T04:18:40+00:00

We have a request for a fast method of getting CPU and NUMA node IDs from userspace. This patch implements a getcpu VDSO function, similar to x86. Ben suggested we use SPRG3 which is userspace readable. SPRG3 can be modified by a KVM guest, so we save the SPRG3 value in the paca and restore it when transitioning from the guest to the host. I have a glibc patch that implements sched_getcpu on top of this. Testing on a POWER7: baseline: 538 cycles vdso: 30 cycles Signed-off-by: Anton Blanchard Signed-off-by: Benjamin Herrenschmidt

Fix common misspellings

2011-03-31T14:26:23+00:00

Fixes generated by 'codespell' and manually reviewed. Signed-off-by: Lucas De Marchi