visualtest/README.txt
author Ryan C. Gordon
Mon, 01 Jan 2018 19:16:51 -0500
changeset 11803 454f6dc9cb85
parent 7924 fcb86d323770
permissions -rwxr-xr-x
windows: Remove references to GetVersionExA (thanks, Andrew Pilley!).

"GetVersionExA is deprecated in windows 8.1 and above's SDK, causing a warning
when building against the win10 SDK. Attached patch cleans up the usage for a
warning-free build.

GetVersionExA was being used to test to see if SDL was running on win9x or
winnt. A quick chat with Ryan on twitter suggested that SDL doesn't
officially support win9x anymore, so the call to this can be outright removed.

As an aside, replacing the call to GetVersionExA with VerifyVersionInfoA (the
recommended path) would have been pointless, as VerifyVersionInfoA only
supports VER_PLATFORM_WIN32_NT and doesn't officially support any other value
for dwPlatformId currently. (And it's probable that win9x SDKs didn't have
VerifyVersionInfo* in them anyway.)"

Fixes Bugzilla #4019.
icculus@7924
     1
/*!
icculus@7924
     2
icculus@7924
     3
\mainpage Visual and Interactive Test Automation for SDL 2.0
icculus@7924
     4
icculus@7924
     5
\section license_sec License
icculus@7924
     6
Check the file \c COPYING.txt for licensing information.
icculus@7924
     7
icculus@7924
     8
\section intro_sec Introduction
icculus@7924
     9
The goal of this GSoC project is to automate the testing of testsprite2.
icculus@7924
    10
testsprite2 takes 26 parameters which have thousands of valid combinations and is
icculus@7924
    11
used to validate SDL's window, mouse and rendering behaviour. By having a test
icculus@7924
    12
harness that runs testsprite2 with various command line argument strings and
icculus@7924
    13
validates the output for each run, we can make testing an easier task for
icculus@7924
    14
maintainers, contributors and testers. The test harness can be used by a continuous
icculus@7924
    15
integration system (like buildbot or jenkins) to validate SDL after checkins.
icculus@7924
    16
icculus@7924
    17
SDL Homepage: http://libsdl.org/
icculus@7924
    18
icculus@7924
    19
\section build_sec Building
icculus@7924
    20
icculus@7924
    21
\subsection build_linux Building on Linux/Cygwin
icculus@7924
    22
<tt>./autogen.sh; ./configure; make;</tt>
icculus@7924
    23
icculus@7924
    24
\subsection build_windows Building on Windows
icculus@7924
    25
Use the Visual Studio solution under \c SDL/VisualC/visualtest.
icculus@7924
    26
icculus@7924
    27
\section docs_sec Documentation
icculus@7924
    28
Documentation is available via Doxygen. To build the documentation, cd to the
icculus@7924
    29
SDL/visualtest/docs directory and run \c doxygen. A good starting point for
icculus@7924
    30
exploring the documentation is \c SDL/visualtest/docs/html/index.html
icculus@7924
    31
icculus@7924
    32
\section usage_sec Usage
icculus@7924
    33
To see all the options supported by the test harness, just run \c testharness
icculus@7924
    34
with no arguments.
icculus@7924
    35
icculus@7924
    36
At the moment the following options are supported:
icculus@7924
    37
\li \c sutapp - Path to the system under test (SUT) application
icculus@7924
    38
\li \c sutargs - Launch the SUT with the specified arguments string
icculus@7924
    39
\li \c timeout - The maximum time after which the SUT process will be killed;
icculus@7924
    40
	passed as hh:mm:ss; default 00:01:00
icculus@7924
    41
\li \c variator - Which variator to use; see \ref variators_sec
icculus@7924
    42
\li \c num-variations - The number of variations to run for; taken to be 
icculus@7924
    43
	1 for the random variator and ALL for the exhaustive variator by default
icculus@7924
    44
\li \c no-launch - Just print the arguments string for each variation without
icculus@7924
    45
	launching the SUT or performing any actions
icculus@7924
    46
\li \c parameter-config - A config file that describes the command line parameters
icculus@7924
    47
	supported by the SUT; see \ref paramconfig_sec or the sample *.parameters files
icculus@7924
    48
	for more details
icculus@7924
    49
\li \c action-config - A config file with a list of actions to be performed while
icculus@7924
    50
	the SUT is running; see \ref actionconfig_sec or the sample *.actions files
icculus@7924
    51
\li \c output-dir - Path to the directory where screenshots should be saved; is
icculus@7924
    52
	created if it doesn't exist; taken to be "./output" by default
icculus@7924
    53
\li \c verify-dir - Path to the directory with the verification images; taken to 
icculus@7924
    54
	be "./verify" by default
icculus@7924
    55
icculus@7924
    56
Paths can be relative or absolute.
icculus@7924
    57
icculus@7924
    58
Alternatively, the options can be passed as a config file for convenience:
icculus@7924
    59
icculus@7924
    60
<tt>testharness \-\-config testsprite2_sample.config</tt>
icculus@7924
    61
icculus@7924
    62
For a sample, take a look at the *.config files in this repository.
icculus@7924
    63
icculus@7924
    64
We can also pass a config file and override certain options as necessary:
icculus@7924
    65
<tt>testharness \-\-config testsprite2_sample.config \-\-num-variations 10</tt>
icculus@7924
    66
icculus@7924
    67
Note: You may find it convenient to copy the SUT executable along with any
icculus@7924
    68
resources to the test harness directory. Also note that testsprite2 and its
icculus@7924
    69
resources (icon.bmp) are automatically copied when using the Visual Studio
icculus@7924
    70
solution.
icculus@7924
    71
icculus@7924
    72
\subsection usageexamples_subsec Usage examples:
icculus@7924
    73
icculus@7924
    74
Passing a custom arguments string:
icculus@7924
    75
<tt>testharness \-\-sutapp testsprite2 \-\-sutargs "\-\-cyclecolor \-\-blend mod
icculus@7924
    76
\-\-iterations 2" \-\-action-config xyz.actions</tt>
icculus@7924
    77
icculus@7924
    78
Using the random variator:
icculus@7924
    79
<tt>testharness \-\-sutapp testsprite2 \-\-variator random \-\-num-variations 5
icculus@7924
    80
\-\-parameter-config xyz.parameters \-\-action-config xyz.actions</tt>
icculus@7924
    81
icculus@7924
    82
\subsection config_subsec Config Files
icculus@7924
    83
Config files are an alternate way to pass parameters to the test harness. We
icculus@7924
    84
describe the paramters in a config file and pass that to the test harness using
icculus@7924
    85
the \-\-config option. The config file consists of lines of the form "x=y" where
icculus@7924
    86
x is an option and y is it's value. For boolean options, we simply give the name
icculus@7924
    87
of the option to indicate that it is to be passed to the testharness.
icculus@7924
    88
icculus@7924
    89
The hash '#' character can be used to start a comment from that point to the end
icculus@7924
    90
of the line.
icculus@7924
    91
icculus@7924
    92
\section paramconfig_sec The SUT Parameters File
icculus@7924
    93
To generate variations we need to describe the parameters the will be passed to
icculus@7924
    94
the SUT. This description is given in a parameters file. Each line of the parameters
icculus@7924
    95
file (except the blank lines) represents one command line option with five
icculus@7924
    96
comma separated fields:
icculus@7924
    97
<tt>name, type, values, required, categories</tt>
icculus@7924
    98
icculus@7924
    99
\li \c name is the name of the option, e.g., \c \-\-cyclecolor.
icculus@7924
   100
\li \c type can have one of three values - integer, boolean and enum.
icculus@7924
   101
\li \c values - for integer options this is the valid range of values the option
icculus@7924
   102
	can take, i.e., [min max]. For enum options this is a list of strings that
icculus@7924
   103
	the option can take, e.g., [val1 val2 val3]. For boolean options this field
icculus@7924
   104
	is ignored.
icculus@7924
   105
\li \c required - true if the option is required, false otherwise.
icculus@7924
   106
\li \c categories - a list of categories that the option belongs to. For example,
icculus@7924
   107
	[video mouse audio]
icculus@7924
   108
icculus@7924
   109
Just like with config files, hash characters can be used to start comments.
icculus@7924
   110
icculus@7924
   111
\subsection additionalnotes_subsec Additional Notes
icculus@7924
   112
icculus@7924
   113
\li If you want to have an option that always takes a certain value, use an enum
icculus@7924
   114
	with only one value.
icculus@7924
   115
\li Currently there isn't any way to turn an option off, i.e., all options will
icculus@7924
   116
	be included in the command line options string that is generated using the
icculus@7924
   117
	config. If you don't want an option to be passed to the SUT, remove it from
icculus@7924
   118
	the config file or comment it out.
icculus@7924
   119
icculus@7924
   120
\section variators_sec Variators
icculus@7924
   121
Variators are the mechanism by which we generate strings of command line arguments
icculus@7924
   122
to test the SUT with. A variator is quite simply an iterator that iterates through
icculus@7924
   123
different variations of command line options. There are two variators supported at
icculus@7924
   124
the moment:
icculus@7924
   125
\li \b Exhaustive - Generate all possible combinations of command line arguments
icculus@7924
   126
	that are valid.
icculus@7924
   127
\li \b Random - Generate a random variation each time the variator is called.
icculus@7924
   128
icculus@7924
   129
As an example, let's try a simple .parameters file:\n
icculus@7924
   130
<tt>
icculus@7924
   131
\-\-blend, enum, [add mod], false, [] \n
icculus@7924
   132
\-\-fullscreen, boolean, [], false, []
icculus@7924
   133
</tt>
icculus@7924
   134
icculus@7924
   135
The exhaustive variator would generate the following four variations:\n
icculus@7924
   136
<tt>
icculus@7924
   137
\-\-blend add \n
icculus@7924
   138
\-\-blend mod \n
icculus@7924
   139
\-\-blend add \-\-fullscreen \n
icculus@7924
   140
\-\-blend mod \-\-fullscreen \n
icculus@7924
   141
</tt>
icculus@7924
   142
icculus@7924
   143
The random variator would simply generate a random variation like the following:\n
icculus@7924
   144
<tt>\-\-blend mod</tt>
icculus@7924
   145
icculus@7924
   146
\section actionconfig_sec The Actions File
icculus@7924
   147
Once the SUT process has been launched, automated testing happens using a mechanism
icculus@7924
   148
called actions. A list of actions is read from a file and each action is performed
icculus@7924
   149
on the SUT process sequentially. Each line in the actions file describes an action.
icculus@7924
   150
The format for an action is <tt>hh:mm:ss ACTION_NAME additional parameters</tt>.
icculus@7924
   151
There are five actions supported at the moment:
icculus@7924
   152
\li \b SCREENSHOT - Takes a screenshot of each window owned by the SUT process. The
icculus@7924
   153
	images are saved as \c [hash]_[i].bmp where \c [hash] is the 32 character long
icculus@7924
   154
	hexadecimal MD5 hash of the arguments string that was passed to the SUT while
icculus@7924
   155
	launching it and \c i is the window number. i = 1 is an exceptional case
icculus@7924
   156
	where the \c _[i] is dropped and the filename is simply \c [hash].bmp\n
icculus@7924
   157
	Note: The screenshots are only of the window's client area.
icculus@7924
   158
\li \b VERIFY - Verifies the screenshots taken by the last SCREENSHOT action by
icculus@7924
   159
	comparing them against a verification image. Each \c [hash]_i.bmp image output
icculus@7924
   160
	by the SCREENSHOT action is compared against a \c [hash].bmp image in the
icculus@7924
   161
	verify-dir.
icculus@7924
   162
\li \b QUIT - Gracefully quits the SUT process. On Windows this means sending a
icculus@7924
   163
	WM_CLOSE message to each window owned by the SUT process. On Linux it means
icculus@7924
   164
	sending a SIGQUIT signal to the SUT process.
icculus@7924
   165
\li \b KILL - Forcefully kills the SUT process. This is useful when the SUT process
icculus@7924
   166
	doesn't respond to the QUIT action.
icculus@7924
   167
\li <b>LAUNCH [/path/to/executable] [args]</b> - Runs an executable with \c [args]
icculus@7924
   168
	as the arguments string.
icculus@7924
   169
icculus@7924
   170
Just like with config files, hash characters can be used to start comments.
icculus@7924
   171
icculus@7924
   172
\section contint_sec Continuous Integration (CI)
icculus@7924
   173
One of the goals of the project was to create a test harness that integrates
icculus@7924
   174
with CI systems to provide automated visual and interactive testing to SDL.
icculus@7924
   175
icculus@7924
   176
At the moment the test harness can be run in two modes that are useful for CI:
icculus@7924
   177
\li Crash testing mode - launch the SUT with every variation and all parameters,
icculus@7924
   178
	report to the CI if there's a crash
icculus@7924
   179
\li Visual testing mode - launch and visually verify the SUT for a smaller subset
icculus@7924
   180
	of the parameters
icculus@7924
   181
icculus@7924
   182
Look at the launch_harness.sh/launch_harness.cmd for an example scripts that run the
icculus@7924
   183
test harness for all variations with all parameters and report an error on a crash.
icculus@7924
   184
The script uses the testsprite2_crashtest config, so remember to copy those files
icculus@7924
   185
over to the test harness executable directory along with the script.
icculus@7924
   186
icculus@7924
   187
\section todo_sec TODOs
icculus@7924
   188
\li Allow specifying a clipping box along with the VERIFY action, i.e., hh:mm:ss
icculus@7924
   189
	VERIFY x, y, w, h
icculus@7924
   190
\li Add support for spaces between the equals sign in test harness config files
icculus@7924
   191
\li Implement the SCREENSHOT action on Linux
icculus@7924
   192
\li Add a pairwise variator
icculus@7924
   193
\li Add actions to inject keyboard/mouse events
icculus@7924
   194
\li Add actions to manipulate the SUT window, e.g., minimize, restore, resize
icculus@7924
   195
\li Add support to load and save screenshots as .pngs instead of .bmps
icculus@7924
   196
icculus@7924
   197
\section issues_sec Known Issues
icculus@7924
   198
\li The QUIT action does not work on a testsprite2 process with multiple windows.
icculus@7924
   199
	This appears to be an issue with testsprite2.
icculus@7924
   200
\li The SCREENSHOT action doesn't capture the testsprite2 window correctly if the
icculus@7924
   201
	--fullscreen option is supplied. It works with --fullscreen-desktop, however.
icculus@7924
   202
icculus@7924
   203
\section moreinfo_sec More Information
icculus@7924
   204
icculus@7924
   205
Author Contact Info:\n
icculus@7924
   206
Apoorv Upreti \c \<apoorvupreti@gmail.com\>
icculus@7924
   207
icculus@7924
   208
Other useful links:
icculus@7924
   209
- Project Repository: https://bitbucket.org/nerdap/sdlvisualtest
icculus@7924
   210
- Project Wiki: https://github.com/nerdap/autotestsprite2/wiki
icculus@7924
   211
- Project Blog: http://nerdap.github.io
icculus@7924
   212
- Verification images for testsprite2_blendmodes: https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip
icculus@7924
   213
- Verification images for testsprite2_geometry: https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip
icculus@7924
   214
*/