pyphosphor/obmc/utils, branch master

pathtree: Make dataitems() use the cache dict under restricted args

2018-05-16T05:27:29+00:00

This achieves a 100x improvement in iteration time when the subtree rooted at the root of the full tree and there is no depth limit (i.e. the caller has requested all "populated" items). Whilst it sounds like a restricted case, it is a very common query by way of phosphor-objmgr's obmc/mapper/server.py:ObjectMapper.process_old_owner() Note this is a quasi API break, as some keys that were previously not present in the results may now appear: Explicitly storing 'None' into the data structure will have the key with 'None' appear in the full-tree dataitems() case. However, given this was filtered previously, no existing callers should be storing 'None' into the tree as they would not have been able to retrieve it via dataitems(). Before: $ python -m obmc.utils.testpathtree ... Iteration tests: depth=1, width=1, n=1000: 0.135853052139 depth=1, width=2, n=1000: 0.203811883926 depth=1, width=3, n=1000: 0.26814198494 depth=1, width=4, n=1000: 0.333888053894 depth=2, width=1, n=1000: 0.193987131119 depth=2, width=2, n=1000: 0.264018058777 depth=2, width=3, n=1000: 0.327262878418 depth=2, width=4, n=1000: 0.38805603981 depth=3, width=1, n=1000: 0.253651857376 depth=3, width=2, n=1000: 0.317117929459 depth=3, width=3, n=1000: 0.385557889938 depth=3, width=4, n=1000: 0.452265024185 depth=4, width=1, n=1000: 0.327889919281 depth=4, width=2, n=1000: 0.390358924866 depth=4, width=3, n=1000: 0.459683895111 depth=4, width=4, n=1000: 0.530153989792 After: $ python -m obmc.utils.testpathtree ... Iteration tests: depth=1, width=1, n=1000: 0.0012412071228 depth=1, width=2, n=1000: 0.00455403327942 depth=1, width=3, n=1000: 0.00307989120483 depth=1, width=4, n=1000: 0.00356507301331 depth=2, width=1, n=1000: 0.00118088722229 depth=2, width=2, n=1000: 0.00169396400452 depth=2, width=3, n=1000: 0.00234699249268 depth=2, width=4, n=1000: 0.00300288200378 depth=3, width=1, n=1000: 0.00100708007812 depth=3, width=2, n=1000: 0.00161695480347 depth=3, width=3, n=1000: 0.00234794616699 depth=3, width=4, n=1000: 0.00315403938293 depth=4, width=1, n=1000: 0.00101804733276 depth=4, width=2, n=1000: 0.00204801559448 depth=4, width=3, n=1000: 0.00281095504761 depth=4, width=4, n=1000: 0.0070219039917 Change-Id: Ice3afd12e2b112227735f0f1dedb6a8ea594740c Signed-off-by: Andrew Jeffery

pathtree: Allow get() on subtrees

2018-05-16T05:27:29+00:00

When running recent PathTree modifications on a BMC we can see failures in the log such as "Failed to invoke ObjectMapper method" from the phosphor-fru-fault-monitor. This can be triggered via the openbmc-test-automation suite, for example with the following command $ robot -v OPENBMC_HOST:w5.aus.stglabs.ibm.com --include \ Create_Test_Error_Callout_And_Verify extended/test_association.robot Capturing the DBus traffic across this test case (`busctl capture > dbus.pcap`) and analysing it with Wireshark showed the following method invocation: ``` 0000 6c 01 00 01 53 00 00 00 16 00 00 00 c6 00 00 00 l...S........... 0010 01 01 6f 00 22 00 00 00 2f 78 79 7a 2f 6f 70 65 ..o.".../xyz/ope 0020 6e 62 6d 63 5f 70 72 6f 6a 65 63 74 2f 6f 62 6a nbmc_project/obj 0030 65 63 74 5f 6d 61 70 70 65 72 00 00 00 00 00 00 ect_mapper...... 0040 03 01 73 00 09 00 00 00 47 65 74 4f 62 6a 65 63 ..s.....GetObjec 0050 74 00 00 00 00 00 00 00 02 01 73 00 20 00 00 00 t.........s. ... 0060 78 79 7a 2e 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a xyz.openbmc_proj 0070 65 63 74 2e 4f 62 6a 65 63 74 4d 61 70 70 65 72 ect.ObjectMapper 0080 00 00 00 00 00 00 00 00 06 01 73 00 20 00 00 00 ..........s. ... 0090 78 79 7a 2e 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a xyz.openbmc_proj 00a0 65 63 74 2e 4f 62 6a 65 63 74 4d 61 70 70 65 72 ect.ObjectMapper 00b0 00 00 00 00 00 00 00 00 08 01 67 00 03 73 61 73 ..........g..sas 00c0 00 00 00 00 00 00 00 00 07 01 73 00 05 00 00 00 ..........s..... 00d0 3a 31 2e 35 37 00 00 00 20 00 00 00 2f 78 79 7a :1.57... .../xyz 00e0 2f 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a 65 63 74 /openbmc_project 00f0 2f 6c 65 64 2f 67 72 6f 75 70 73 2f 00 00 00 00 /led/groups/.... 0100 27 00 00 00 22 00 00 00 6f 72 67 2e 66 72 65 65 '..."...org.free 0110 64 65 73 6b 74 6f 70 2e 44 42 75 73 2e 4f 62 6a desktop.DBus.Obj 0120 65 63 74 4d 61 6e 61 67 65 72 00 ectManager. ``` and the response ``` 0000 6c 03 01 01 3f 00 00 00 9d 08 00 00 5e 00 00 00 l...?.......^... 0010 06 01 73 00 05 00 00 00 3a 31 2e 35 37 00 00 00 ..s.....:1.57... 0020 04 01 73 00 27 00 00 00 6f 72 67 2e 66 72 65 65 ..s.'...org.free 0030 64 65 73 6b 74 6f 70 2e 44 42 75 73 2e 45 72 72 desktop.DBus.Err 0040 6f 72 2e 46 69 6c 65 4e 6f 74 46 6f 75 6e 64 00 or.FileNotFound. 0050 05 01 75 00 16 00 00 00 08 01 67 00 01 73 00 00 ..u.......g..s.. 0060 07 01 73 00 05 00 00 00 3a 31 2e 31 35 00 00 00 ..s.....:1.15... 0070 3a 00 00 00 70 61 74 68 20 6f 72 20 6f 62 6a 65 :...path or obje 0080 63 74 20 6e 6f 74 20 66 6f 75 6e 64 3a 20 2f 78 ct not found: /x 0090 79 7a 2f 6f 70 65 6e 62 6d 63 5f 70 72 6f 6a 65 yz/openbmc_proje 00a0 63 74 2f 6c 65 64 2f 67 72 6f 75 70 73 2f 00 ct/led/groups/. ``` These packets translate to the following `busctl` invocation and response: ``` root@witherspoon:~# busctl call \ xyz.openbmc_project.ObjectMapper \ /xyz/openbmc_project/object_mapper \ xyz.openbmc_project.ObjectMapper \ GetObject sas \ /xyz/openbmc_project/led/groups/ \ 1 \ org.freedesktop.Dbus.ObjectManager path or object not found: /xyz/openbmc_project/led/groups/ root@witherspoon:~# ``` From `busctl tree` we can see that the objects do actually appear on the bus: root@witherspoon:~# busctl tree ... Service xyz.openbmc_project.LED.GroupManager: └─/xyz └─/xyz/openbmc_project └─/xyz/openbmc_project/led └─/xyz/openbmc_project/led/groups ├─/xyz/openbmc_project/led/groups/bmc_booted ├─/xyz/openbmc_project/led/groups/bmc_fault ├─/xyz/openbmc_project/led/groups/boxelder_fault ├─/xyz/openbmc_project/led/groups/core0_fault ├─/xyz/openbmc_project/led/groups/core10_fault ... ``` However, the /xyz/openbmc_project/led/groups/ path represents a subtree root, not a leaf element. I didn't realise this was a supported use-case, and over-aggressively removed support for it in the recent patch adding the flat dict cache. Add in a fallback to iterating the PathTree datastructure when we take a cache miss. Change-Id: I658c3fdd6b87be9d814c59a654e63ec5bb83e2bd Fixes: 52aeb314c0eb ("pathtree: Cache keys in a flat dict for fast lookup") Signed-off-by: Andrew Jeffery

testpathtree: Add dataitems() stress test

2018-05-14T00:38:18+00:00

Change-Id: I7493a8d2d9df6123a3d80f17b2ba268608997066 Signed-off-by: Andrew Jeffery

pathtree: Cache keys in a flat dict for fast lookup

2018-05-14T00:38:18+00:00

This knocks about a second off of the depth stress tests: $ python -m obmc.utils.testpathtree Depth tests: 1: 5.18097305298 2: 6.18765211105 3: 7.3427259922 4: 8.33903598785 5: 9.55236506462 6: 10.7165560722 7: 12.1563079357 8: 13.3570129871 9: 14.7827298641 10: 16.0967490673 Width tests: 1: 1.20649981499 2: 1.20889282227 4: 1.20902085304 8: 1.22063994408 16: 1.22446990013 32: 1.21649098396 64: 1.21024799347 128: 1.21360588074 256: 1.22187900543 512: 1.23233604431 1024: 1.21607708931 2048: 1.21069407463 4096: 1.22389698029 8192: 1.20828580856 16384: 1.21290493011 32768: 1.21552395821 65536: 1.23201704025 131072: 1.21459794044 262144: 1.24190402031 524288: 1.21342992783 1048576: 1.21248602867 However, we get just over a 300% improvement in the host reboot scenario, which is captured by some separate tools used to analyse the PathTree's performance issues: $ # Before $ ./replay.py -n 1000 ptinit.log ptreboot.log deep copy: 1000 iterations took 3.0744523939938517s ops script: 1000 iterations took 10.03650910500437s net: 1000 iterations took 6.9620567110105185s $ # After $ ./replay.py -n 1000 ptinit.log ptreboot.log deep copy: 1000 iterations took 6.784729664999759s ops script: 1000 iterations took 9.0453162470003s net: 1000 iterations took 2.260586582000542s Change-Id: I6826b99950fa1561d292908c4ffff1137ecaa1bc Signed-off-by: Andrew Jeffery

testpathtree: Add stress tests

2018-05-14T00:38:18+00:00

Indicative performance on a lightly loaded 3rd Gen Lenovo X1 Carbon: $ # Timing 1,000,000 executions of the test case, time units are seconds $ python -m obmc.utils.testpathtree Depth tests: 1: 6.01530885696 2: 7.11315083504 3: 8.41325497627 4: 9.51180887222 5: 10.7866010666 6: 12.0375499725 7: 13.4880149364 8: 14.7702541351 9: 16.3227319717 10: 17.6581590176 Width tests: 1: 1.19656515121 2: 1.20851492882 4: 1.2031879425 8: 1.19995284081 16: 1.21298193932 32: 1.20965600014 64: 1.20766687393 128: 1.21984100342 256: 1.21412611008 512: 1.21589684486 1024: 1.2073469162 2048: 1.2244079113 4096: 1.21275901794 8192: 1.22486519814 16384: 1.21715903282 32768: 1.21435189247 65536: 1.2201769352 131072: 1.21944999695 262144: 1.21434497833 524288: 1.20947313309 1048576: 1.21527695656 Change-Id: I05aa3fb59ddc2510e22e22a8802e166e2e801bab Signed-off-by: Andrew Jeffery

utils: Add characterisation tests for PathTree

2018-05-14T00:38:18+00:00

Change-Id: I1ad4832779e85a54a75b3551a8beeba429679e1c Signed-off-by: Andrew Jeffery

utils: Remove ListMatch

2018-04-05T02:47:23+00:00

This is a pointless class given python lambdas. It was also broken with aea38c65. Drop. Change-Id: Ib269fee1b09cd04defe6ec98c2f5dc4118d5425e Signed-off-by: Brad Bishop

Enable pycodestyle

2018-03-29T19:44:10+00:00

Apply fix-ups and run pycodestyle during CI testing. Change-Id: I1005495b11e228abdc8d40a51dbf81c4a6e6c92c Signed-off-by: Brad Bishop

Port to python 3

2018-03-12T18:34:00+00:00

Port code to python 3 yet remain backward compatible to python 2. This port is in preparation for yocto 2.4 upgrade. Resolves openbmc/openbmc#2920 Change-Id: Ie06d206a3de3de2ea636b043c099a0185b32ce53 Signed-off-by: CamVan Nguyen

Revert "Port obmc to python3.5"

2018-01-31T20:56:22+00:00

Parse error with this change: TypeError: iter() returned non-iterator of type 'PathTreeItemIterator' This reverts commit f99783be6592130bbad2d0f69ada2b7ac61618f3. Change-Id: Id2b7f4af5284d25d59e19905c7b873726b02f700 Signed-off-by: Brad Bishop