test, tools: introduce tbot README

introduce a README how to use tbot for testing U-Boot
and/or linux kernels.

Signed-off-by: Heiko Schocher <hs@denx.de>

1 files changed, 185 insertions, 0 deletions

diff --git a/tools/tbot/README b/tools/tbot/README
new file mode 100644
index 0000000000..a637a63d32
--- /dev/null
+++ b/tools/tbot/README
@@ -0,0 +1,185 @@
+# Copyright (c) 2016 DENX Software Engineering GmbH
+# Heiko Schocher <hs@denx.de>
+#
+# SPDX-License-Identifier:	GPL-2.0+
+#
+
+What is tbot ?
+==============
+
+tbot is a tool for executing testcases on boards.
+Source code found on [1]
+Based on DUTS [2]
+written in python
+
+Basic Ideas of tbot
+===================
+(see also the figure:
+https://github.com/hsdenx/tbot/blob/master/doc/tbot_structure.png )
+
+- Virtual laboratory (VL)
+   VL is the basic environment that groups:
+  - [a number of] boards - target devices on which tbot executes testcases.
+  - one Lab PC
+
+- Test case (TC):
+  A piece of python code, which uses the tbot class from [1].
+  Tbot provides functions for sending shell commands and parsing the
+  shell commands output.
+  Tbot waits endless for a shell commands end (detected through reading
+  the consoles prompt).
+  A TC can also call other TC-es.
+
+  remark:
+  Tbot not really waits endless, for a shell commands end, instead
+  tbot starts a watchdog in the background, and if it triggers, tbot
+  ends the TC as failed. In the tbot beginning there was a lot of
+  timeouts / retry cases, but it turned out, that waiting endless
+  is robust and easy ...
+
+- Host PC (where tbot runs, currently only linux host tested)
+  must not a powerful machine (For example [3], I use a
+  raspberry pi for running tbot and buildbot)
+
+- Lab PC:
+  - Host PC connects through ssh to the Lab PC
+    -> so it is possible to test boards, which
+       are not at the same place as the Host PC.
+       (Lab PC and Host PC can be the same of course)
+       -> maybe we can setup a Testsystem, which does nightly
+          U-Boot/Linux builds and test from current mainline U-Boot
+          on boards wherever they are accessible.
+
+  - necessary tasks a Lab PC must deliver:
+    - connect to boards console through a shell command.
+    - power on/off boards through a shell command
+    - detect the current power state of a board through
+      a shell command
+
+  - optional tasks:
+    - tftp server (for example loading images)
+    - nfs server (used as rootfs for linux kernels)
+    - Internet access for example for downloading
+      U-Boot source with git.
+    - toolchains installed for compiling source code
+
+      -> a linux machine is preffered.
+
+  - currently only Lab PC with an installed linux supported/tested.
+
+- Boards(s):
+  the boards on which shell commands are executed.
+
+- Board state:
+  equals to the software, the board is currently running.
+
+  Currently tbot supports 2 board states:
+    - "u-boot", if the board is running U-Boot
+    - "linux", if the board is running a linux kernel
+
+  It should be easy to add other board states to tbot, see
+  https://github.com/hsdenx/tbot/tree/master/src/lab_api/state_[u-boot/linux].py
+
+  A board state is detected through analysing the boards
+  shell prompt. In linux, tbot sets a special tbot prompt,
+  in U-Boot the prompt is static, and configurable in tbot through
+  a board config file.
+
+  A TC can say in which board state it want to send shell commands.
+  Tbot tries to detect the current board state, if board is not in
+  the requested  board state, tbot tries to switch into the correct
+  state. If this fails, the TC fails.
+
+  It is possible to switch in a single TC between board states.
+
+- tbot cmdline parameters:
+
+$ python2.7 src/common/tbot.py --help
+Usage: tbot.py [options]
+
+Options:
+  -h, --help            show this help message and exit
+  -c CFGFILE, --cfgfile=CFGFILE
+                        the tbot common configfilename
+  -l LOGFILE, --logfile=LOGFILE
+                        the tbot logfilename, if default, tbot creates a
+                        defaultnamelogfile
+  -t TC, --testcase=TC  the testcase which should be run
+  -v, --verbose         be verbose, print all read/write to stdout
+  -w WORKDIR, --workdir=WORKDIR
+                        set workdir, default os.getcwd()
+$
+
+tbot needs the following files for proper execution:
+
+  - tbot board configuration file (option -c):
+    A board configuration file contains settings tbot needs to
+    connect to the Lab PC and board specific variable settings
+    for testcases.
+
+  - name of the logfile tbot creates (option -l)
+    defaultname: 'log/' + now.strftime("%Y-%m-%d-%H-%M") + '.log'
+
+  - tbots working directory (option -w)
+
+  - the testcasename tbot executes (option -t)
+
+You are interested and want to use tbot?
+If so, please read on the file:
+tools/tbot/README.install
+
+If not read [3] ;-)
+
+Heiko Schocher <hs@denx.de>
+v1 2016.01.22
+
+--------------
+[1] https://github.com/hsdenx/tbot
+[2] http://www.denx.de/wiki/DUTS/DUTSDocs
+[3] automated Testsetup with buildbot and tbot doing cyclic tests
+    (buildbot used for starting tbot TC and web presentation of the
+     results, all testing done through tbot):
+    http://xeidos.ddns.net/buildbot/tgrid
+    Host PC in Letkes/hungary
+    VL in munich/germany
+
+    Fancy things are done here, for example:
+    - http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/43/steps/shell/logs/tbotlog
+      (I try to cleanup the logfile soon, so it is not so filled with crap ;-)
+      A first step see here:
+      http://xeidos.ddns.net/buildbot/builders/smartweb_dfu/builds/45/steps/shell/logs/tbotlog
+      (same TC now with the new loglevel = 'CON' ... not yet perfect)
+      Executed steps:
+      - clone u-boot.git
+      - set toolchain
+      - get a list of patchwork patches from my U-Boots ToDo list
+      - download all of them, and check them with checkpatch
+        and apply them to u-boot.git
+      - compile U-Boot for the smartweb board
+      - install the resulting images on the smartweb board
+      - boot U-boot
+      - test DFU
+      - more TC should be added here for testing U-Boot
+
+    - automatic "git bisect"
+      https://github.com/hsdenx/tbot/blob/master/src/tc/tc_board_git_bisect.py
+      http://xeidos.ddns.net/buildbot/builders/tqm5200s/builds/3/steps/shell/logs/tbotlog
+
+      If a current U-Boot image not works on the tqm5200 board
+      this TC can be started. It starts a "git bisect" session,
+      and compiles for each step U-Boot, install it on the tqm5200
+      board, and tests if U-Boot works !
+
+      At the end, it detects the commit, which breaks the board
+
+      This TC is not dependend on U-Boot nor on a special board. It
+      needs only 3 variables:
+      tb.board_git_bisect_get_source_tc: TC which gets the source tree, in which
+        "git bisect" should be executed
+      tb.board_git_bisect_call_tc: TC which gets called every "git bisect" step,
+        which executes commands for detecting if current source code is OK or not.
+        This could be a TC which compiles U-Boot, install it on the board and
+        executes TC on the new booted U-Boot image. ! Board maybe gets borken,
+        as not all U-Boot images work, so you must have a TC which install U-Boot
+        image for example through a debugger.
+      tb.board_git_bisect_good_commit: last nown good commit id