visualtest/README.txt
changeset 7924 fcb86d323770
     1.1 --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
     1.2 +++ b/visualtest/README.txt	Sun Nov 10 00:32:23 2013 -0500
     1.3 @@ -0,0 +1,214 @@
     1.4 +/*!
     1.5 +
     1.6 +\mainpage Visual and Interactive Test Automation for SDL 2.0
     1.7 +
     1.8 +\section license_sec License
     1.9 +Check the file \c COPYING.txt for licensing information.
    1.10 +
    1.11 +\section intro_sec Introduction
    1.12 +The goal of this GSoC project is to automate the testing of testsprite2.
    1.13 +testsprite2 takes 26 parameters which have thousands of valid combinations and is
    1.14 +used to validate SDL's window, mouse and rendering behaviour. By having a test
    1.15 +harness that runs testsprite2 with various command line argument strings and
    1.16 +validates the output for each run, we can make testing an easier task for
    1.17 +maintainers, contributors and testers. The test harness can be used by a continuous
    1.18 +integration system (like buildbot or jenkins) to validate SDL after checkins.
    1.19 +
    1.20 +SDL Homepage: http://libsdl.org/
    1.21 +
    1.22 +\section build_sec Building
    1.23 +
    1.24 +\subsection build_linux Building on Linux/Cygwin
    1.25 +<tt>./autogen.sh; ./configure; make;</tt>
    1.26 +
    1.27 +\subsection build_windows Building on Windows
    1.28 +Use the Visual Studio solution under \c SDL/VisualC/visualtest.
    1.29 +
    1.30 +\section docs_sec Documentation
    1.31 +Documentation is available via Doxygen. To build the documentation, cd to the
    1.32 +SDL/visualtest/docs directory and run \c doxygen. A good starting point for
    1.33 +exploring the documentation is \c SDL/visualtest/docs/html/index.html
    1.34 +
    1.35 +\section usage_sec Usage
    1.36 +To see all the options supported by the test harness, just run \c testharness
    1.37 +with no arguments.
    1.38 +
    1.39 +At the moment the following options are supported:
    1.40 +\li \c sutapp - Path to the system under test (SUT) application
    1.41 +\li \c sutargs - Launch the SUT with the specified arguments string
    1.42 +\li \c timeout - The maximum time after which the SUT process will be killed;
    1.43 +	passed as hh:mm:ss; default 00:01:00
    1.44 +\li \c variator - Which variator to use; see \ref variators_sec
    1.45 +\li \c num-variations - The number of variations to run for; taken to be 
    1.46 +	1 for the random variator and ALL for the exhaustive variator by default
    1.47 +\li \c no-launch - Just print the arguments string for each variation without
    1.48 +	launching the SUT or performing any actions
    1.49 +\li \c parameter-config - A config file that describes the command line parameters
    1.50 +	supported by the SUT; see \ref paramconfig_sec or the sample *.parameters files
    1.51 +	for more details
    1.52 +\li \c action-config - A config file with a list of actions to be performed while
    1.53 +	the SUT is running; see \ref actionconfig_sec or the sample *.actions files
    1.54 +\li \c output-dir - Path to the directory where screenshots should be saved; is
    1.55 +	created if it doesn't exist; taken to be "./output" by default
    1.56 +\li \c verify-dir - Path to the directory with the verification images; taken to 
    1.57 +	be "./verify" by default
    1.58 +
    1.59 +Paths can be relative or absolute.
    1.60 +
    1.61 +Alternatively, the options can be passed as a config file for convenience:
    1.62 +
    1.63 +<tt>testharness \-\-config testsprite2_sample.config</tt>
    1.64 +
    1.65 +For a sample, take a look at the *.config files in this repository.
    1.66 +
    1.67 +We can also pass a config file and override certain options as necessary:
    1.68 +<tt>testharness \-\-config testsprite2_sample.config \-\-num-variations 10</tt>
    1.69 +
    1.70 +Note: You may find it convenient to copy the SUT executable along with any
    1.71 +resources to the test harness directory. Also note that testsprite2 and its
    1.72 +resources (icon.bmp) are automatically copied when using the Visual Studio
    1.73 +solution.
    1.74 +
    1.75 +\subsection usageexamples_subsec Usage examples:
    1.76 +
    1.77 +Passing a custom arguments string:
    1.78 +<tt>testharness \-\-sutapp testsprite2 \-\-sutargs "\-\-cyclecolor \-\-blend mod
    1.79 +\-\-iterations 2" \-\-action-config xyz.actions</tt>
    1.80 +
    1.81 +Using the random variator:
    1.82 +<tt>testharness \-\-sutapp testsprite2 \-\-variator random \-\-num-variations 5
    1.83 +\-\-parameter-config xyz.parameters \-\-action-config xyz.actions</tt>
    1.84 +
    1.85 +\subsection config_subsec Config Files
    1.86 +Config files are an alternate way to pass parameters to the test harness. We
    1.87 +describe the paramters in a config file and pass that to the test harness using
    1.88 +the \-\-config option. The config file consists of lines of the form "x=y" where
    1.89 +x is an option and y is it's value. For boolean options, we simply give the name
    1.90 +of the option to indicate that it is to be passed to the testharness.
    1.91 +
    1.92 +The hash '#' character can be used to start a comment from that point to the end
    1.93 +of the line.
    1.94 +
    1.95 +\section paramconfig_sec The SUT Parameters File
    1.96 +To generate variations we need to describe the parameters the will be passed to
    1.97 +the SUT. This description is given in a parameters file. Each line of the parameters
    1.98 +file (except the blank lines) represents one command line option with five
    1.99 +comma separated fields:
   1.100 +<tt>name, type, values, required, categories</tt>
   1.101 +
   1.102 +\li \c name is the name of the option, e.g., \c \-\-cyclecolor.
   1.103 +\li \c type can have one of three values - integer, boolean and enum.
   1.104 +\li \c values - for integer options this is the valid range of values the option
   1.105 +	can take, i.e., [min max]. For enum options this is a list of strings that
   1.106 +	the option can take, e.g., [val1 val2 val3]. For boolean options this field
   1.107 +	is ignored.
   1.108 +\li \c required - true if the option is required, false otherwise.
   1.109 +\li \c categories - a list of categories that the option belongs to. For example,
   1.110 +	[video mouse audio]
   1.111 +
   1.112 +Just like with config files, hash characters can be used to start comments.
   1.113 +
   1.114 +\subsection additionalnotes_subsec Additional Notes
   1.115 +
   1.116 +\li If you want to have an option that always takes a certain value, use an enum
   1.117 +	with only one value.
   1.118 +\li Currently there isn't any way to turn an option off, i.e., all options will
   1.119 +	be included in the command line options string that is generated using the
   1.120 +	config. If you don't want an option to be passed to the SUT, remove it from
   1.121 +	the config file or comment it out.
   1.122 +
   1.123 +\section variators_sec Variators
   1.124 +Variators are the mechanism by which we generate strings of command line arguments
   1.125 +to test the SUT with. A variator is quite simply an iterator that iterates through
   1.126 +different variations of command line options. There are two variators supported at
   1.127 +the moment:
   1.128 +\li \b Exhaustive - Generate all possible combinations of command line arguments
   1.129 +	that are valid.
   1.130 +\li \b Random - Generate a random variation each time the variator is called.
   1.131 +
   1.132 +As an example, let's try a simple .parameters file:\n
   1.133 +<tt>
   1.134 +\-\-blend, enum, [add mod], false, [] \n
   1.135 +\-\-fullscreen, boolean, [], false, []
   1.136 +</tt>
   1.137 +
   1.138 +The exhaustive variator would generate the following four variations:\n
   1.139 +<tt>
   1.140 +\-\-blend add \n
   1.141 +\-\-blend mod \n
   1.142 +\-\-blend add \-\-fullscreen \n
   1.143 +\-\-blend mod \-\-fullscreen \n
   1.144 +</tt>
   1.145 +
   1.146 +The random variator would simply generate a random variation like the following:\n
   1.147 +<tt>\-\-blend mod</tt>
   1.148 +
   1.149 +\section actionconfig_sec The Actions File
   1.150 +Once the SUT process has been launched, automated testing happens using a mechanism
   1.151 +called actions. A list of actions is read from a file and each action is performed
   1.152 +on the SUT process sequentially. Each line in the actions file describes an action.
   1.153 +The format for an action is <tt>hh:mm:ss ACTION_NAME additional parameters</tt>.
   1.154 +There are five actions supported at the moment:
   1.155 +\li \b SCREENSHOT - Takes a screenshot of each window owned by the SUT process. The
   1.156 +	images are saved as \c [hash]_[i].bmp where \c [hash] is the 32 character long
   1.157 +	hexadecimal MD5 hash of the arguments string that was passed to the SUT while
   1.158 +	launching it and \c i is the window number. i = 1 is an exceptional case
   1.159 +	where the \c _[i] is dropped and the filename is simply \c [hash].bmp\n
   1.160 +	Note: The screenshots are only of the window's client area.
   1.161 +\li \b VERIFY - Verifies the screenshots taken by the last SCREENSHOT action by
   1.162 +	comparing them against a verification image. Each \c [hash]_i.bmp image output
   1.163 +	by the SCREENSHOT action is compared against a \c [hash].bmp image in the
   1.164 +	verify-dir.
   1.165 +\li \b QUIT - Gracefully quits the SUT process. On Windows this means sending a
   1.166 +	WM_CLOSE message to each window owned by the SUT process. On Linux it means
   1.167 +	sending a SIGQUIT signal to the SUT process.
   1.168 +\li \b KILL - Forcefully kills the SUT process. This is useful when the SUT process
   1.169 +	doesn't respond to the QUIT action.
   1.170 +\li <b>LAUNCH [/path/to/executable] [args]</b> - Runs an executable with \c [args]
   1.171 +	as the arguments string.
   1.172 +
   1.173 +Just like with config files, hash characters can be used to start comments.
   1.174 +
   1.175 +\section contint_sec Continuous Integration (CI)
   1.176 +One of the goals of the project was to create a test harness that integrates
   1.177 +with CI systems to provide automated visual and interactive testing to SDL.
   1.178 +
   1.179 +At the moment the test harness can be run in two modes that are useful for CI:
   1.180 +\li Crash testing mode - launch the SUT with every variation and all parameters,
   1.181 +	report to the CI if there's a crash
   1.182 +\li Visual testing mode - launch and visually verify the SUT for a smaller subset
   1.183 +	of the parameters
   1.184 +
   1.185 +Look at the launch_harness.sh/launch_harness.cmd for an example scripts that run the
   1.186 +test harness for all variations with all parameters and report an error on a crash.
   1.187 +The script uses the testsprite2_crashtest config, so remember to copy those files
   1.188 +over to the test harness executable directory along with the script.
   1.189 +
   1.190 +\section todo_sec TODOs
   1.191 +\li Allow specifying a clipping box along with the VERIFY action, i.e., hh:mm:ss
   1.192 +	VERIFY x, y, w, h
   1.193 +\li Add support for spaces between the equals sign in test harness config files
   1.194 +\li Implement the SCREENSHOT action on Linux
   1.195 +\li Add a pairwise variator
   1.196 +\li Add actions to inject keyboard/mouse events
   1.197 +\li Add actions to manipulate the SUT window, e.g., minimize, restore, resize
   1.198 +\li Add support to load and save screenshots as .pngs instead of .bmps
   1.199 +
   1.200 +\section issues_sec Known Issues
   1.201 +\li The QUIT action does not work on a testsprite2 process with multiple windows.
   1.202 +	This appears to be an issue with testsprite2.
   1.203 +\li The SCREENSHOT action doesn't capture the testsprite2 window correctly if the
   1.204 +	--fullscreen option is supplied. It works with --fullscreen-desktop, however.
   1.205 +
   1.206 +\section moreinfo_sec More Information
   1.207 +
   1.208 +Author Contact Info:\n
   1.209 +Apoorv Upreti \c \<apoorvupreti@gmail.com\>
   1.210 +
   1.211 +Other useful links:
   1.212 +- Project Repository: https://bitbucket.org/nerdap/sdlvisualtest
   1.213 +- Project Wiki: https://github.com/nerdap/autotestsprite2/wiki
   1.214 +- Project Blog: http://nerdap.github.io
   1.215 +- Verification images for testsprite2_blendmodes: https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip
   1.216 +- Verification images for testsprite2_geometry: https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip
   1.217 +*/