| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For opencapi, we currently do impedance calibration when initializing
the PHY for the device, which could run in parallel if we were rich
and had multiple opencapi devices. But if 2 devices are on the same
obus, the 2 calibration sequences could overlap, which likely yields
bad results and is useless anyway since it only needs to be done once
per obus.
This patch splits the opencapi PHY reset in 2 parts:
- a 'init' part called serially at boot. That's when zcal is done. If
we have 2 devices on the same socket, the zcal won't be redone,
since we're called serially and we'll see it has already be done for
the obus
- a 'reset' part called during fundamental reset as a prereq for link
training. It does the PHY setup for a set of lanes and the dccal.
The PHY team confirmed there's no dependency between zcal and the
other reset steps and it can be moved earlier.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If two opencapi adapters are on the same obus, we may try to train the
two links in parallel at boot time, when all the PCI links are being
trained. Both links use the same i2c controller to handle the reset
signal, so some care is needed to make sure resetting one doesn't
interfere with the reset of the other. We need to keep track of the
current state of the i2c controller (and use locking).
This went mostly unnoticed as you need to have 2 opencapi cards on the
same socket and links tended to train anyway because of the retries.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Give more time to the FPGA to process the reset signal. The previous
delay, 5ms, is too short for newer adapters with bigger FPGAs. Extend
it to 250ms.
Ultimately, that delay will likely end up being added to the opencapi
specification, but we are not there yet.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We haven't hit any problem so far, but from the ODL designer, the ODL
should be in reset when it is enabled.
The ODL remains in reset until we start a fundamental reset to
initiate link training. We still assert and deassert the ODL reset
signal as part of the normal procedure just before training the
link. Asserting is therefore useless at boot, since the ODL is already
in reset, but we keep it as it's only a scom write and it's needed
when we reset/retrain from the OS.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Split the function to assert and deassert the reset signal on the ODL,
so that we can keep the ODL in reset while we reset the adapter,
therefore having a window where both sides are in reset.
It is actually not required with our current DLx at boot time, but I
need to split the ODL reset function for the following patch and it
will become useful/required later when we introduce resetting an
opencapi link from the OS.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
| |
This is really to avoid confusion with a later patch and clarify
whether we're resetting the ODL or the adapter.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It's possible to set up performance counters for the PLL to detect
various conditions for the links in nvlink or opencapi mode. Since
those counters are currently unused, let's configure them when an obus
is in opencapi mode to detect CRC errors on the link. Each link has
two counters:
- CRC error detected by the host
- CRC error detected by the DLx (NAK received by the host)
We also dump the counters shortly after the link trains, but they can
be read multiple times through cronus, pdbg or linux. The counters are
configured to be reset after each read.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
ODL registers used to control the opencapi link state have an address
built on a base address and an offset for each brick which can be
computed instead of hard-coded individually for each brick.
Rework how we access the ODL registers, to avoid repeating switch
statements all over the place.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
| |
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
| |
This cache is written but never read. Wiring it up would gain us little
(except added complexity), and it obviously hasn't been missed thus far,
so remove it altogether.
Signed-off-by: Reza Arbab <arbab@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
If the link trains in degraded mode, log the ODL endpoint information
register for debug. Its content is specific to the DLx and TLx
implementation, so this is really information useful for the hardware
team.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
| |
There's no status readily available to tell the effective link
width. Instead, we have to look at the individual status of each lane,
on the transmit and receive direction. All relevant information is in
the ODL status register.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
| |
Log the link training status register in case of failure to train.
It can have useful information for the hardware team.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In platform_ocapi, we define i2c_{reset,presence}_odl{0,1} to specify the
appropriate reset/presence GPIO pins for devices connected to ODL0 and ODL1
respectively.
This is obviously wrong, because a device connected to brick 2 and a device
connected to brick 4 are going to be different devices connected to
different I2C pins, but rather conveniently we haven't had to deal with
systems that can use the full 4 bricks as yet. Now that we're adding
OpenCAPI support for Witherspoon, we should change this to specify pins
separately for all 4 bricks.
Replace i2c_{reset,presence}_odl{0,1} with
i2c_{reset,presence}_brick{2,3,4,5} and update the presence detection code,
device reset code, and existing platforms accordingly.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Reviewed-by: Oliver O'Halloran <oohall@gmail.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There is no standardised way to determine the presence and type of devices
connected to an NPU on POWER9.
Currently, we hardcode device types based on platform type (as no platform
currently supports both OpenCAPI and NVLink), and for OpenCAPI platforms
we use I2C to detect presence.
Witherspoon (and potentially other platforms later on) supports both
NVLink and OpenCAPI, and additionally uses SXM2 connectors which can carry
more than one link, rather than the SlimSAS connectors used for OpenCAPI on
Zaius and ZZ. This necessitates some special handling.
Add a platform callback for NPU device detection. In a later patch, we
will use this to implement Witherspoon-specific device detection. For now,
add a Witherspoon stub that sets all links to NVLink (i.e. current
behaviour).
Move the existing I2C-based presence detection for OpenCAPI devices on
Zaius/ZZ into common code, which we use by default for platforms which do
not define a callback. Clean up the use of the ibm,npu-link-type property,
which will now be exposed solely for debugging and not consumed internally.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Replace probe_npu2() and probe_npu2_opencapi() with a new shared
probe_npu2(). Refactor some of the common NPU setup code into shared code.
No functional change. This patch does not implement support for using both
types of devices simultaneously on the same NPU - we expect to add this
sometime in the future.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Acked-by: Reza Arbab <arbab@linux.ibm.com>
Reviewed-by: Alistair Popple <alistair@popple.id.au>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
On Witherspoon, OpenCAPI devices attached to link indexes 0 and 1 are
handled by bricks 2 and 3.
Rename index to brick_index, and add a new field, link_index, to
refer to the link index. For now, we set those values identically.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Acked-by: Reza Arbab <arbab@linux.ibm.com>
Reviewed-by: Alistair Popple <alistair@popple.id.au>
Reviewed-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In opal_npu_tl_set(), we made a typo that means the OPAL_NPU_TL_SET call
may not clear the enable bits for templates that were previously enabled
but are now disabled.
Fix the typo so we clear NPU2_OTL_CONFIG1_TX_TEMP2_EN as well as
TEMP{1,3}_EN.
Reported-by: Tyler Seredynski <tseredynski@gmail.com>
Fixes: cd8b82a8e83ed ("npu2-opencapi: Add OpenCAPI OPAL API calls")
Cc: stable
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Even if an opencapi link is down, we currently always try to issue a
config read operation when probing for PCI devices, because of the
default scan map used for an opencapi PHB. The config operation fails,
as expected, but it can also raise a FIR bit and trigger an HMI.
For opencapi, there's no root device like for a "normal" PCI PHB, so
there's no reason to do the config operation. To fix it, we keep the
scan map blank by default, and only add a device once the link is
trained.
CC: stable # v6.1+
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
device tree
Currently, we distinguish between NPU links for NVLink devices and OpenCAPI
devices through the use of two different compatible strings - ibm,npu-link
and ibm,npu-link-opencapi.
As we move towards supporting configurations with both NVLink and OpenCAPI
devices behind a single NPU, we need to detect the device type as part of
presence detection, which can't happen until well after the point where the
HDAT or platform code has created the NPU device tree nodes. Changing a
node's compatible string after it's been created is a bit ugly, so instead
we should move the device type to a new property which we can add to the
node later on.
Get rid of the ibm,npu-link-opencapi compatible string, add a new
ibm,npu-link-type property, and a helper function to check the link type.
Add an "unknown" device type in preparation for later patches to detect
device type dynamically.
These device tree bindings are entirely internal to skiboot and are not
consumed directly by Linux, so this shouldn't break anything (other than
internal BML lab environments).
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We've been reliably training the opencapi link on the first attempt
for quite a while. Furthermore, if it doesn't train on the first
attempt, retries haven't been that useful. So let's reduce the number
of attempts we do to train the link.
2 retries = 3 attempts to train.
Each (failed) training sequence costs about 3 seconds.
Signed-off-by: Frederic Barrat <fbarrat@linux.ibm.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The PHB callback 'get_link_state' is always reporting the link width,
irrespective of the link status and even when the link is down. It is
causing too much work (and failures) when the PHB is probed during pci
init.
The fix is to look at the link status first and report the link as
down when appropriate.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now that links may train in parallel, traces shown during training can
be all mixed up. So add a prefix to all the traces to clearly identify
the chip and link the trace refers to:
OCAPI[<chip id>:<link id>]: this is a very useful message
The lower-level hardware procedures (npu2-hw-procedures.c) also print
traces which would need work. But that code is being reworked to be
better integrated with opencapi and nvidia, so leave it alone for now.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reorder our link training steps so that they are executed on
fundamental reset instead of during the initial setup. Skiboot always
call a fundamental reset on all the PHBs during pci init.
It is done through a state machine, similarly to what is done for
'real' PHBs.
This is the first step for a longer term goal to be able to trigger an
adapter reset from linux. We'll need the reset callbacks of the PHB to
be defined. We have to handle the various delays differently, since a
linux thread shouldn't stay stuck waiting in opal for too long.
No functional changes.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Rework a bit the code to reset the opencapi adapter:
- make clearer which i2c pin is resetting which device
- break the reset operation in smaller chunks. This is really to
prepare for a future patch.
No functional changes.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Presence detection is not part of the opencapi specification. So each
platform may choose to implement it the way it wants.
All current platforms implement it through an i2c device where we can
query a pin to know if a device is connected or not. ZZ and Zaius have
a similar design and even use the same i2c information and pin
numbers.
However, presence detection on older ZZ planar (older than v4) doesn't
work, so we don't activate it for now, until our lab systems are
upgraded and it's better tested.
Presence detection on witherspoon is still being worked on. It's
shaping up to be quite different, so we may have to revisit the topic
in a later patch.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Reviewed-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When setting up an opencapi link, we set the transport muxes first,
then set the PHY training config register, which includes disabling
nvlink mode for the bricks. That's the order of the init sequence, as
found in the NPU workbook.
In reality, doing so works, but it raises 2 FIR bits in the PowerBus
OLL FIR Register for the 2 links when we configure the transport
muxes. Presumably because nvlink is not disabled yet and we are
configuring the transport muxes for opencapi.
bit 60: link0 internal error
bit 61: link1 internal error
Overall the current setup ends up being correct and everything works,
but we raise 2 FIR bits.
So tweak the order of operations to disable nvlink before configuring
the transport muxes. Incidentally, this is what the scripts from the
opencapi enablement team were doing all along.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When we setup a link, we always enable ODL0 and ODL1 at the same time
in the PHY training config register, even though we are setting up
only one OTL/ODL, so it raises a "link internal error" FIR bit in the
PowerBus OLL FIR Register for the second link. The error is harmless,
as we'll eventually setup the second link, but there's no reason to
raise that FIR bit.
The fix is simply to only enable the ODL we are using for the link.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.ibm.com>
|
| |
|
|
|
|
| |
Fixes: CID 264267
Signed-off-by: Cyril Bur <cyril.bur@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.vnet.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We don't support resetting an opencapi link yet.
Commit fe6d86b9 ("pci: Make fast reboot creset PHBs in parallel")
tries resetting any PHB whose slot defines a 'run_sm' callback. It
raises an assert when applied to an opencapi PHB, as 'run_sm' calls
the 'freset' callback, which is not yet defined for opencapi.
Fix it for now by removing the currently useless definition of
'run_sm' on the opencapi slot. It will print a message in the skiboot
log because the PHB cannot be reset, which is correct. It will all go
away when we add support for resetting an opencapi link.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Acked-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.vnet.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add three OPAL API calls that are required by the ocxl driver.
- OPAL_NPU_SPA_SETUP
The Shared Process Area (SPA) is a table containing one entry (a
"Process Element") per memory context which can be accessed by the
OpenCAPI device.
- OPAL_NPU_SPA_CLEAR_CACHE
The NPU keeps a cache of recently accessed memory contexts. When a
Process Element is removed from the SPA, the cache for the link must be
cleared.
- OPAL_NPU_TL_SET
The Transaction Layer specification defines several templates for
messages to be exchanged on the link. During link setup, the host and
device must negotiate what templates are supported on both sides and at
what rates those messages can be sent.
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.vnet.ibm.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Scan the OpenCAPI links under the NPU, and for each link, reset the card,
set up a device, train the link and register a PHB.
Implement the necessary operations for the OpenCAPI PHB type.
For bringup, test and debug purposes, we allow an NVRAM setting,
"opencapi-link-training" that can be set to either disable link training
completely or to use the prbs31 test pattern.
To disable link training:
nvram -p ibm,skiboot --update-config opencapi-link-training=none
To use prbs31:
nvram -p ibm,skiboot --update-config opencapi-link-training=prbs31
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.vnet.ibm.com>
|
|
|
Scan the device tree for NPUs with OpenCAPI links and configure the NPU per
the initialisation sequence in the NPU OpenCAPI workbook.
Training of individual links and setup of per-AFU/link configuration will
be in a later patch.
Signed-off-by: Andrew Donnellan <andrew.donnellan@au1.ibm.com>
Signed-off-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Reviewed-by: Frederic Barrat <fbarrat@linux.vnet.ibm.com>
Signed-off-by: Stewart Smith <stewart@linux.vnet.ibm.com>
|
