Introduce NPU2 support

NVLink2 is a new feature introduced on POWER9 systems. It is an
evolution of of the NVLink1 feature included in POWER8+ systems but
adds several new features including support for GPU address
translation using the Nest MMU and cache coherence.

Similar to NVLink1 the functionality is exposed to the OS as a series
of virtual PCIe devices. However the actual hardware interfaces are
significantly different which limits the amount of common code that
can be shared between implementations in the firmware.

This patch adds basic hardware initialisation and exposure of the
virtual NVLink2 PCIe devices to the running OS.

Signed-off-by: Alistair Popple <alistair@popple.id.au>
Signed-off-by: Stewart Smith <stewart@linux.vnet.ibm.com>

2 files changed, 43 insertions, 6 deletions

diff --git a/doc/device-tree/nvlink.rst b/doc/device-tree/nvlink.rst
index 8375dc4e..6ce44e9d 100644
--- a/doc/device-tree/nvlink.rst
+++ b/doc/device-tree/nvlink.rst
@@ -66,6 +66,27 @@ NPU bindings:
     };
   };
 
+GPU memory bindings
+-------------------
+
+.. code-block:: dts
+
+        memory@100000000 {
+                device_type = "memory"
+                compatible = "ibm,coherent-device-memory";
+                linux,usable-memory = <0x0 0x100000000 0x0 0x0>;
+
+        ; denotes a region of unplugged system memory
+
+                reg = <0x0 0x100000000 0x0 0x80000000>;
+                ibm,associativity = <0x4 0x0 0x0 0x0 0x64>;
+
+        ; numa associativity for the memory once it is hotplugged
+
+                phandle = <0x10000abc>;
+                linux,phandle = <0x10000abc>;
+        };
+
 Emulated PCI device bindings
 ----------------------------
 
@@ -83,7 +104,8 @@ Emulated PCI device bindings
                         ibm,pci-config-space-type = <0x1>;
                         vendor-id = <0x1014>;
                         ibm,gpu = <0x100002f7>; /* phandle pointing the associated GPU PCI device node */
-  	  	        phandle = <0x100002fc>;
+			memory-region = <0x10000abc>; /* phandle pointing to the GPU memory */
+			phandle = <0x100002fc>;
                 };
 
                 pci@1 {
@@ -94,6 +116,7 @@ Emulated PCI device bindings
                         ibm,pci-config-space-type = <0x1>;
                         vendor-id = <0x1014>;
                         ibm,gpu = <0x100002f5>;
+                        memory-region = <0x10000def>;
                         phandle = <0x100002fe>;
                         class-code = <0x60400>;
                         linux,phandle = <0x100002fe>;
@@ -107,6 +130,7 @@ Emulated PCI device bindings
                         ibm,pci-config-space-type = <0x1>;
                         vendor-id = <0x1014>;
                         ibm,gpu = <0x100002f7>;
+                        memory-region = <0x10000abc>;
                         phandle = <0x100002fd>;
                         class-code = <0x60400>;
                         linux,phandle = <0x100002fd>;
@@ -120,6 +144,7 @@ Emulated PCI device bindings
                         ibm,pci-config-space-type = <0x1>;
                         vendor-id = <0x1014>;
                         ibm,gpu = <0x100002f5>;
+                        memory-region = <0x10000def>;
                         phandle = <0x100002ff>;
                         class-code = <0x60400>;
                         linux,phandle = <0x100002ff>;
diff --git a/doc/nvlink.rst b/doc/nvlink.rst
index d0359687..cba64df5 100644
--- a/doc/nvlink.rst
+++ b/doc/nvlink.rst
@@ -38,14 +38,15 @@ related to the setup of DMA windows.
 Configuration Space Parameters
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-============ =============== =====
-============ =============== =====
-Vendor ID    0x1014          (IBM)
+============ =================== =====
+============ =================== =====
+Vendor ID    0x1014              (IBM)
 Device ID    0x04ea
 Revision ID  0x00
-Class        0x068000        (Bridge Device Other, ProgIf = 0x0)
+Class        0x068000 / 0x068001 (Bridge Device Other, ProgIf = 0x0 / 0x1)
 BAR0/1       TL/DL Registers
-============ =============== =====
+BAR2/3       GEN-ID Registers    (Only for rev-id = 0x1)
+============ =================== =====
 
 TL/DL Registers
 ^^^^^^^^^^^^^^^
@@ -59,6 +60,17 @@ to 64-bit BAR#0 of the emulated PCI device configuration space. ::
       	     | DL (64K)  |
  BAR#0	     +-----------+
 
+Generation Registers (GEN-ID)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+On POWER9 each link has 64K of generation ID registers for the relaxed
+ordering mode syncronisation. Refer to the programming guide for
+details of the register layout in this BAR.
+
+Relaxed ordering mode will be disabled by default as it requires
+device driver support. Device drivers will need to request relaxed
+ordering mode through some yet to be designed mechanism.
+
 Vendor Specific Capabilities
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 ::