author Alex Szpakowski Tue, 05 May 2015 19:01:55 -0300 changeset 9587 18979eac0a35 parent 7924 fcb86d323770 permissions -rwxr-xr-x
Replaced all remaining uses of NSAutoreleasePool with @autoreleasepool blocks (bugzilla #2680.)
     1 /*!

     2

     3 \mainpage Visual and Interactive Test Automation for SDL 2.0

     4

     5 \section license_sec License

     6 Check the file \c COPYING.txt for licensing information.

     7

     8 \section intro_sec Introduction

     9 The goal of this GSoC project is to automate the testing of testsprite2.

    10 testsprite2 takes 26 parameters which have thousands of valid combinations and is

    11 used to validate SDL's window, mouse and rendering behaviour. By having a test

    12 harness that runs testsprite2 with various command line argument strings and

    13 validates the output for each run, we can make testing an easier task for

    14 maintainers, contributors and testers. The test harness can be used by a continuous

    15 integration system (like buildbot or jenkins) to validate SDL after checkins.

    16

    17 SDL Homepage: http://libsdl.org/

    18

    19 \section build_sec Building

    20

    21 \subsection build_linux Building on Linux/Cygwin

    22 <tt>./autogen.sh; ./configure; make;</tt>

    23

    24 \subsection build_windows Building on Windows

    25 Use the Visual Studio solution under \c SDL/VisualC/visualtest.

    26

    27 \section docs_sec Documentation

    28 Documentation is available via Doxygen. To build the documentation, cd to the

    29 SDL/visualtest/docs directory and run \c doxygen. A good starting point for

    30 exploring the documentation is \c SDL/visualtest/docs/html/index.html

    31

    32 \section usage_sec Usage

    33 To see all the options supported by the test harness, just run \c testharness

    34 with no arguments.

    35

    36 At the moment the following options are supported:

    37 \li \c sutapp - Path to the system under test (SUT) application

    38 \li \c sutargs - Launch the SUT with the specified arguments string

    39 \li \c timeout - The maximum time after which the SUT process will be killed;

    40 	passed as hh:mm:ss; default 00:01:00

    41 \li \c variator - Which variator to use; see \ref variators_sec

    42 \li \c num-variations - The number of variations to run for; taken to be

    43 	1 for the random variator and ALL for the exhaustive variator by default

    44 \li \c no-launch - Just print the arguments string for each variation without

    45 	launching the SUT or performing any actions

    46 \li \c parameter-config - A config file that describes the command line parameters

    47 	supported by the SUT; see \ref paramconfig_sec or the sample *.parameters files

    48 	for more details

    49 \li \c action-config - A config file with a list of actions to be performed while

    50 	the SUT is running; see \ref actionconfig_sec or the sample *.actions files

    51 \li \c output-dir - Path to the directory where screenshots should be saved; is

    52 	created if it doesn't exist; taken to be "./output" by default

    53 \li \c verify-dir - Path to the directory with the verification images; taken to

    54 	be "./verify" by default

    55

    56 Paths can be relative or absolute.

    57

    58 Alternatively, the options can be passed as a config file for convenience:

    59

    60 <tt>testharness \-\-config testsprite2_sample.config</tt>

    61

    62 For a sample, take a look at the *.config files in this repository.

    63

    64 We can also pass a config file and override certain options as necessary:

    65 <tt>testharness \-\-config testsprite2_sample.config \-\-num-variations 10</tt>

    66

    67 Note: You may find it convenient to copy the SUT executable along with any

    68 resources to the test harness directory. Also note that testsprite2 and its

    69 resources (icon.bmp) are automatically copied when using the Visual Studio

    70 solution.

    71

    72 \subsection usageexamples_subsec Usage examples:

    73

    74 Passing a custom arguments string:

    75 <tt>testharness \-\-sutapp testsprite2 \-\-sutargs "\-\-cyclecolor \-\-blend mod

    76 \-\-iterations 2" \-\-action-config xyz.actions</tt>

    77

    78 Using the random variator:

    79 <tt>testharness \-\-sutapp testsprite2 \-\-variator random \-\-num-variations 5

    80 \-\-parameter-config xyz.parameters \-\-action-config xyz.actions</tt>

    81

    82 \subsection config_subsec Config Files

    83 Config files are an alternate way to pass parameters to the test harness. We

    84 describe the paramters in a config file and pass that to the test harness using

    85 the \-\-config option. The config file consists of lines of the form "x=y" where

    86 x is an option and y is it's value. For boolean options, we simply give the name

    87 of the option to indicate that it is to be passed to the testharness.

    88

    89 The hash '#' character can be used to start a comment from that point to the end

    90 of the line.

    91

    92 \section paramconfig_sec The SUT Parameters File

    93 To generate variations we need to describe the parameters the will be passed to

    94 the SUT. This description is given in a parameters file. Each line of the parameters

    95 file (except the blank lines) represents one command line option with five

    96 comma separated fields:

    97 <tt>name, type, values, required, categories</tt>

    98

    99 \li \c name is the name of the option, e.g., \c \-\-cyclecolor.

   100 \li \c type can have one of three values - integer, boolean and enum.

   101 \li \c values - for integer options this is the valid range of values the option

   102 	can take, i.e., [min max]. For enum options this is a list of strings that

   103 	the option can take, e.g., [val1 val2 val3]. For boolean options this field

   104 	is ignored.

   105 \li \c required - true if the option is required, false otherwise.

   106 \li \c categories - a list of categories that the option belongs to. For example,

   107 	[video mouse audio]

   108

   109 Just like with config files, hash characters can be used to start comments.

   110

   111 \subsection additionalnotes_subsec Additional Notes

   112

   113 \li If you want to have an option that always takes a certain value, use an enum

   114 	with only one value.

   115 \li Currently there isn't any way to turn an option off, i.e., all options will

   116 	be included in the command line options string that is generated using the

   117 	config. If you don't want an option to be passed to the SUT, remove it from

   118 	the config file or comment it out.

   119

   120 \section variators_sec Variators

   121 Variators are the mechanism by which we generate strings of command line arguments

   122 to test the SUT with. A variator is quite simply an iterator that iterates through

   123 different variations of command line options. There are two variators supported at

   124 the moment:

   125 \li \b Exhaustive - Generate all possible combinations of command line arguments

   126 	that are valid.

   127 \li \b Random - Generate a random variation each time the variator is called.

   128

   129 As an example, let's try a simple .parameters file:\n

   130 <tt>

   131 \-\-blend, enum, [add mod], false, [] \n

   132 \-\-fullscreen, boolean, [], false, []

   133 </tt>

   134

   135 The exhaustive variator would generate the following four variations:\n

   136 <tt>

   137 \-\-blend add \n

   138 \-\-blend mod \n

   139 \-\-blend add \-\-fullscreen \n

   140 \-\-blend mod \-\-fullscreen \n

   141 </tt>

   142

   143 The random variator would simply generate a random variation like the following:\n

   144 <tt>\-\-blend mod</tt>

   145

   146 \section actionconfig_sec The Actions File

   147 Once the SUT process has been launched, automated testing happens using a mechanism

   148 called actions. A list of actions is read from a file and each action is performed

   149 on the SUT process sequentially. Each line in the actions file describes an action.

   150 The format for an action is <tt>hh:mm:ss ACTION_NAME additional parameters</tt>.

   151 There are five actions supported at the moment:

   152 \li \b SCREENSHOT - Takes a screenshot of each window owned by the SUT process. The

   153 	images are saved as \c [hash]_[i].bmp where \c [hash] is the 32 character long

   154 	hexadecimal MD5 hash of the arguments string that was passed to the SUT while

   155 	launching it and \c i is the window number. i = 1 is an exceptional case

   156 	where the \c _[i] is dropped and the filename is simply \c [hash].bmp\n

   157 	Note: The screenshots are only of the window's client area.

   158 \li \b VERIFY - Verifies the screenshots taken by the last SCREENSHOT action by

   159 	comparing them against a verification image. Each \c [hash]_i.bmp image output

   160 	by the SCREENSHOT action is compared against a \c [hash].bmp image in the

   161 	verify-dir.

   162 \li \b QUIT - Gracefully quits the SUT process. On Windows this means sending a

   163 	WM_CLOSE message to each window owned by the SUT process. On Linux it means

   164 	sending a SIGQUIT signal to the SUT process.

   165 \li \b KILL - Forcefully kills the SUT process. This is useful when the SUT process

   166 	doesn't respond to the QUIT action.

   167 \li <b>LAUNCH [/path/to/executable] [args]</b> - Runs an executable with \c [args]

   168 	as the arguments string.

   169

   170 Just like with config files, hash characters can be used to start comments.

   171

   172 \section contint_sec Continuous Integration (CI)

   173 One of the goals of the project was to create a test harness that integrates

   174 with CI systems to provide automated visual and interactive testing to SDL.

   175

   176 At the moment the test harness can be run in two modes that are useful for CI:

   177 \li Crash testing mode - launch the SUT with every variation and all parameters,

   178 	report to the CI if there's a crash

   179 \li Visual testing mode - launch and visually verify the SUT for a smaller subset

   180 	of the parameters

   181

   182 Look at the launch_harness.sh/launch_harness.cmd for an example scripts that run the

   183 test harness for all variations with all parameters and report an error on a crash.

   184 The script uses the testsprite2_crashtest config, so remember to copy those files

   185 over to the test harness executable directory along with the script.

   186

   187 \section todo_sec TODOs

   188 \li Allow specifying a clipping box along with the VERIFY action, i.e., hh:mm:ss

   189 	VERIFY x, y, w, h

   190 \li Add support for spaces between the equals sign in test harness config files

   191 \li Implement the SCREENSHOT action on Linux

   192 \li Add a pairwise variator

   193 \li Add actions to inject keyboard/mouse events

   194 \li Add actions to manipulate the SUT window, e.g., minimize, restore, resize

   195 \li Add support to load and save screenshots as .pngs instead of .bmps

   196

   197 \section issues_sec Known Issues

   198 \li The QUIT action does not work on a testsprite2 process with multiple windows.

   199 	This appears to be an issue with testsprite2.

   200 \li The SCREENSHOT action doesn't capture the testsprite2 window correctly if the

   201 	--fullscreen option is supplied. It works with --fullscreen-desktop, however.

   202

   203 \section moreinfo_sec More Information

   204

   205 Author Contact Info:\n

   206 Apoorv Upreti \c \<apoorvupreti@gmail.com\>

   207

   208 Other useful links:

   209 - Project Repository: https://bitbucket.org/nerdap/sdlvisualtest

   210 - Project Wiki: https://github.com/nerdap/autotestsprite2/wiki

   211 - Project Blog: http://nerdap.github.io

   212 - Verification images for testsprite2_blendmodes: https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip

   213 - Verification images for testsprite2_geometry: https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip

   214 */