visualtest/README.txt
author Sylvain Becker <sylvain.becker@gmail.com>
Tue, 13 Aug 2019 16:00:08 +0200
changeset 13008 8e299f937b6c
parent 7924 fcb86d323770
permissions -rw-r--r--
Android: SDL_image/SDL_mixer/SDL_ttf partially compiling with CMake (bug 3918)
icculus@7924
     1
/*!
icculus@7924
     2
icculus@7924
     3
\mainpage Visual and Interactive Test Automation for SDL 2.0
icculus@7924
     4
icculus@7924
     5
\section license_sec License
icculus@7924
     6
Check the file \c COPYING.txt for licensing information.
icculus@7924
     7
icculus@7924
     8
\section intro_sec Introduction
icculus@7924
     9
The goal of this GSoC project is to automate the testing of testsprite2.
icculus@7924
    10
testsprite2 takes 26 parameters which have thousands of valid combinations and is
icculus@7924
    11
used to validate SDL's window, mouse and rendering behaviour. By having a test
icculus@7924
    12
harness that runs testsprite2 with various command line argument strings and
icculus@7924
    13
validates the output for each run, we can make testing an easier task for
icculus@7924
    14
maintainers, contributors and testers. The test harness can be used by a continuous
icculus@7924
    15
integration system (like buildbot or jenkins) to validate SDL after checkins.
icculus@7924
    16
icculus@7924
    17
SDL Homepage: http://libsdl.org/
icculus@7924
    18
icculus@7924
    19
\section build_sec Building
icculus@7924
    20
icculus@7924
    21
\subsection build_linux Building on Linux/Cygwin
icculus@7924
    22
<tt>./autogen.sh; ./configure; make;</tt>
icculus@7924
    23
icculus@7924
    24
\subsection build_windows Building on Windows
icculus@7924
    25
Use the Visual Studio solution under \c SDL/VisualC/visualtest.
icculus@7924
    26
icculus@7924
    27
\section docs_sec Documentation
icculus@7924
    28
Documentation is available via Doxygen. To build the documentation, cd to the
icculus@7924
    29
SDL/visualtest/docs directory and run \c doxygen. A good starting point for
icculus@7924
    30
exploring the documentation is \c SDL/visualtest/docs/html/index.html
icculus@7924
    31
icculus@7924
    32
\section usage_sec Usage
icculus@7924
    33
To see all the options supported by the test harness, just run \c testharness
icculus@7924
    34
with no arguments.
icculus@7924
    35
icculus@7924
    36
At the moment the following options are supported:
icculus@7924
    37
\li \c sutapp - Path to the system under test (SUT) application
icculus@7924
    38
\li \c sutargs - Launch the SUT with the specified arguments string
icculus@7924
    39
\li \c timeout - The maximum time after which the SUT process will be killed;
icculus@7924
    40
	passed as hh:mm:ss; default 00:01:00
icculus@7924
    41
\li \c variator - Which variator to use; see \ref variators_sec
icculus@7924
    42
\li \c num-variations - The number of variations to run for; taken to be 
icculus@7924
    43
	1 for the random variator and ALL for the exhaustive variator by default
icculus@7924
    44
\li \c no-launch - Just print the arguments string for each variation without
icculus@7924
    45
	launching the SUT or performing any actions
icculus@7924
    46
\li \c parameter-config - A config file that describes the command line parameters
icculus@7924
    47
	supported by the SUT; see \ref paramconfig_sec or the sample *.parameters files
icculus@7924
    48
	for more details
icculus@7924
    49
\li \c action-config - A config file with a list of actions to be performed while
icculus@7924
    50
	the SUT is running; see \ref actionconfig_sec or the sample *.actions files
icculus@7924
    51
\li \c output-dir - Path to the directory where screenshots should be saved; is
icculus@7924
    52
	created if it doesn't exist; taken to be "./output" by default
icculus@7924
    53
\li \c verify-dir - Path to the directory with the verification images; taken to 
icculus@7924
    54
	be "./verify" by default
icculus@7924
    55
icculus@7924
    56
Paths can be relative or absolute.
icculus@7924
    57
icculus@7924
    58
Alternatively, the options can be passed as a config file for convenience:
icculus@7924
    59
icculus@7924
    60
<tt>testharness \-\-config testsprite2_sample.config</tt>
icculus@7924
    61
icculus@7924
    62
For a sample, take a look at the *.config files in this repository.
icculus@7924
    63
icculus@7924
    64
We can also pass a config file and override certain options as necessary:
icculus@7924
    65
<tt>testharness \-\-config testsprite2_sample.config \-\-num-variations 10</tt>
icculus@7924
    66
icculus@7924
    67
Note: You may find it convenient to copy the SUT executable along with any
icculus@7924
    68
resources to the test harness directory. Also note that testsprite2 and its
icculus@7924
    69
resources (icon.bmp) are automatically copied when using the Visual Studio
icculus@7924
    70
solution.
icculus@7924
    71
icculus@7924
    72
\subsection usageexamples_subsec Usage examples:
icculus@7924
    73
icculus@7924
    74
Passing a custom arguments string:
icculus@7924
    75
<tt>testharness \-\-sutapp testsprite2 \-\-sutargs "\-\-cyclecolor \-\-blend mod
icculus@7924
    76
\-\-iterations 2" \-\-action-config xyz.actions</tt>
icculus@7924
    77
icculus@7924
    78
Using the random variator:
icculus@7924
    79
<tt>testharness \-\-sutapp testsprite2 \-\-variator random \-\-num-variations 5
icculus@7924
    80
\-\-parameter-config xyz.parameters \-\-action-config xyz.actions</tt>
icculus@7924
    81
icculus@7924
    82
\subsection config_subsec Config Files
icculus@7924
    83
Config files are an alternate way to pass parameters to the test harness. We
icculus@7924
    84
describe the paramters in a config file and pass that to the test harness using
icculus@7924
    85
the \-\-config option. The config file consists of lines of the form "x=y" where
icculus@7924
    86
x is an option and y is it's value. For boolean options, we simply give the name
icculus@7924
    87
of the option to indicate that it is to be passed to the testharness.
icculus@7924
    88
icculus@7924
    89
The hash '#' character can be used to start a comment from that point to the end
icculus@7924
    90
of the line.
icculus@7924
    91
icculus@7924
    92
\section paramconfig_sec The SUT Parameters File
icculus@7924
    93
To generate variations we need to describe the parameters the will be passed to
icculus@7924
    94
the SUT. This description is given in a parameters file. Each line of the parameters
icculus@7924
    95
file (except the blank lines) represents one command line option with five
icculus@7924
    96
comma separated fields:
icculus@7924
    97
<tt>name, type, values, required, categories</tt>
icculus@7924
    98
icculus@7924
    99
\li \c name is the name of the option, e.g., \c \-\-cyclecolor.
icculus@7924
   100
\li \c type can have one of three values - integer, boolean and enum.
icculus@7924
   101
\li \c values - for integer options this is the valid range of values the option
icculus@7924
   102
	can take, i.e., [min max]. For enum options this is a list of strings that
icculus@7924
   103
	the option can take, e.g., [val1 val2 val3]. For boolean options this field
icculus@7924
   104
	is ignored.
icculus@7924
   105
\li \c required - true if the option is required, false otherwise.
icculus@7924
   106
\li \c categories - a list of categories that the option belongs to. For example,
icculus@7924
   107
	[video mouse audio]
icculus@7924
   108
icculus@7924
   109
Just like with config files, hash characters can be used to start comments.
icculus@7924
   110
icculus@7924
   111
\subsection additionalnotes_subsec Additional Notes
icculus@7924
   112
icculus@7924
   113
\li If you want to have an option that always takes a certain value, use an enum
icculus@7924
   114
	with only one value.
icculus@7924
   115
\li Currently there isn't any way to turn an option off, i.e., all options will
icculus@7924
   116
	be included in the command line options string that is generated using the
icculus@7924
   117
	config. If you don't want an option to be passed to the SUT, remove it from
icculus@7924
   118
	the config file or comment it out.
icculus@7924
   119
icculus@7924
   120
\section variators_sec Variators
icculus@7924
   121
Variators are the mechanism by which we generate strings of command line arguments
icculus@7924
   122
to test the SUT with. A variator is quite simply an iterator that iterates through
icculus@7924
   123
different variations of command line options. There are two variators supported at
icculus@7924
   124
the moment:
icculus@7924
   125
\li \b Exhaustive - Generate all possible combinations of command line arguments
icculus@7924
   126
	that are valid.
icculus@7924
   127
\li \b Random - Generate a random variation each time the variator is called.
icculus@7924
   128
icculus@7924
   129
As an example, let's try a simple .parameters file:\n
icculus@7924
   130
<tt>
icculus@7924
   131
\-\-blend, enum, [add mod], false, [] \n
icculus@7924
   132
\-\-fullscreen, boolean, [], false, []
icculus@7924
   133
</tt>
icculus@7924
   134
icculus@7924
   135
The exhaustive variator would generate the following four variations:\n
icculus@7924
   136
<tt>
icculus@7924
   137
\-\-blend add \n
icculus@7924
   138
\-\-blend mod \n
icculus@7924
   139
\-\-blend add \-\-fullscreen \n
icculus@7924
   140
\-\-blend mod \-\-fullscreen \n
icculus@7924
   141
</tt>
icculus@7924
   142
icculus@7924
   143
The random variator would simply generate a random variation like the following:\n
icculus@7924
   144
<tt>\-\-blend mod</tt>
icculus@7924
   145
icculus@7924
   146
\section actionconfig_sec The Actions File
icculus@7924
   147
Once the SUT process has been launched, automated testing happens using a mechanism
icculus@7924
   148
called actions. A list of actions is read from a file and each action is performed
icculus@7924
   149
on the SUT process sequentially. Each line in the actions file describes an action.
icculus@7924
   150
The format for an action is <tt>hh:mm:ss ACTION_NAME additional parameters</tt>.
icculus@7924
   151
There are five actions supported at the moment:
icculus@7924
   152
\li \b SCREENSHOT - Takes a screenshot of each window owned by the SUT process. The
icculus@7924
   153
	images are saved as \c [hash]_[i].bmp where \c [hash] is the 32 character long
icculus@7924
   154
	hexadecimal MD5 hash of the arguments string that was passed to the SUT while
icculus@7924
   155
	launching it and \c i is the window number. i = 1 is an exceptional case
icculus@7924
   156
	where the \c _[i] is dropped and the filename is simply \c [hash].bmp\n
icculus@7924
   157
	Note: The screenshots are only of the window's client area.
icculus@7924
   158
\li \b VERIFY - Verifies the screenshots taken by the last SCREENSHOT action by
icculus@7924
   159
	comparing them against a verification image. Each \c [hash]_i.bmp image output
icculus@7924
   160
	by the SCREENSHOT action is compared against a \c [hash].bmp image in the
icculus@7924
   161
	verify-dir.
icculus@7924
   162
\li \b QUIT - Gracefully quits the SUT process. On Windows this means sending a
icculus@7924
   163
	WM_CLOSE message to each window owned by the SUT process. On Linux it means
icculus@7924
   164
	sending a SIGQUIT signal to the SUT process.
icculus@7924
   165
\li \b KILL - Forcefully kills the SUT process. This is useful when the SUT process
icculus@7924
   166
	doesn't respond to the QUIT action.
icculus@7924
   167
\li <b>LAUNCH [/path/to/executable] [args]</b> - Runs an executable with \c [args]
icculus@7924
   168
	as the arguments string.
icculus@7924
   169
icculus@7924
   170
Just like with config files, hash characters can be used to start comments.
icculus@7924
   171
icculus@7924
   172
\section contint_sec Continuous Integration (CI)
icculus@7924
   173
One of the goals of the project was to create a test harness that integrates
icculus@7924
   174
with CI systems to provide automated visual and interactive testing to SDL.
icculus@7924
   175
icculus@7924
   176
At the moment the test harness can be run in two modes that are useful for CI:
icculus@7924
   177
\li Crash testing mode - launch the SUT with every variation and all parameters,
icculus@7924
   178
	report to the CI if there's a crash
icculus@7924
   179
\li Visual testing mode - launch and visually verify the SUT for a smaller subset
icculus@7924
   180
	of the parameters
icculus@7924
   181
icculus@7924
   182
Look at the launch_harness.sh/launch_harness.cmd for an example scripts that run the
icculus@7924
   183
test harness for all variations with all parameters and report an error on a crash.
icculus@7924
   184
The script uses the testsprite2_crashtest config, so remember to copy those files
icculus@7924
   185
over to the test harness executable directory along with the script.
icculus@7924
   186
icculus@7924
   187
\section todo_sec TODOs
icculus@7924
   188
\li Allow specifying a clipping box along with the VERIFY action, i.e., hh:mm:ss
icculus@7924
   189
	VERIFY x, y, w, h
icculus@7924
   190
\li Add support for spaces between the equals sign in test harness config files
icculus@7924
   191
\li Implement the SCREENSHOT action on Linux
icculus@7924
   192
\li Add a pairwise variator
icculus@7924
   193
\li Add actions to inject keyboard/mouse events
icculus@7924
   194
\li Add actions to manipulate the SUT window, e.g., minimize, restore, resize
icculus@7924
   195
\li Add support to load and save screenshots as .pngs instead of .bmps
icculus@7924
   196
icculus@7924
   197
\section issues_sec Known Issues
icculus@7924
   198
\li The QUIT action does not work on a testsprite2 process with multiple windows.
icculus@7924
   199
	This appears to be an issue with testsprite2.
icculus@7924
   200
\li The SCREENSHOT action doesn't capture the testsprite2 window correctly if the
icculus@7924
   201
	--fullscreen option is supplied. It works with --fullscreen-desktop, however.
icculus@7924
   202
icculus@7924
   203
\section moreinfo_sec More Information
icculus@7924
   204
icculus@7924
   205
Author Contact Info:\n
icculus@7924
   206
Apoorv Upreti \c \<apoorvupreti@gmail.com\>
icculus@7924
   207
icculus@7924
   208
Other useful links:
icculus@7924
   209
- Project Repository: https://bitbucket.org/nerdap/sdlvisualtest
icculus@7924
   210
- Project Wiki: https://github.com/nerdap/autotestsprite2/wiki
icculus@7924
   211
- Project Blog: http://nerdap.github.io
icculus@7924
   212
- Verification images for testsprite2_blendmodes: https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip
icculus@7924
   213
- Verification images for testsprite2_geometry: https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip
icculus@7924
   214
*/