{"payload":{"allShortcutsEnabled":false,"path":"visualtest","repo":{"id":330008801,"defaultBranch":"main","name":"SDL","ownerLogin":"libsdl-org","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2021-01-15T19:55:54.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/77683370?v=4","public":true,"private":false,"isOrgOwned":true},"currentUser":null,"refInfo":{"name":"f0c061d141d014031c0962a828344fdfc4c91c32","listCacheKey":"v0:1709678518.0","canEdit":false,"refType":"tree","currentOid":"f0c061d141d014031c0962a828344fdfc4c91c32"},"tree":{"items":[{"name":"configs","path":"visualtest/configs","contentType":"directory"},{"name":"docs","path":"visualtest/docs","contentType":"directory"},{"name":"include","path":"visualtest/include","contentType":"directory"},{"name":"src","path":"visualtest/src","contentType":"directory"},{"name":"unittest","path":"visualtest/unittest","contentType":"directory"},{"name":"COPYING.txt","path":"visualtest/COPYING.txt","contentType":"file"},{"name":"Makefile.in","path":"visualtest/Makefile.in","contentType":"file"},{"name":"README.txt","path":"visualtest/README.txt","contentType":"file"},{"name":"acinclude.m4","path":"visualtest/acinclude.m4","contentType":"file"},{"name":"autogen.sh","path":"visualtest/autogen.sh","contentType":"file"},{"name":"compile","path":"visualtest/compile","contentType":"file"},{"name":"config.h","path":"visualtest/config.h","contentType":"file"},{"name":"config.h.in","path":"visualtest/config.h.in","contentType":"file"},{"name":"configure","path":"visualtest/configure","contentType":"file"},{"name":"configure.in","path":"visualtest/configure.in","contentType":"file"},{"name":"depcomp","path":"visualtest/depcomp","contentType":"file"},{"name":"install-sh","path":"visualtest/install-sh","contentType":"file"},{"name":"launch_harness.cmd","path":"visualtest/launch_harness.cmd","contentType":"file"},{"name":"launch_harness.sh","path":"visualtest/launch_harness.sh","contentType":"file"},{"name":"missing","path":"visualtest/missing","contentType":"file"},{"name":"stamp-h1","path":"visualtest/stamp-h1","contentType":"file"},{"name":"testsprite2_sample.actions","path":"visualtest/testsprite2_sample.actions","contentType":"file"},{"name":"testsprite2_sample.config","path":"visualtest/testsprite2_sample.config","contentType":"file"},{"name":"testsprite2_sample.parameters","path":"visualtest/testsprite2_sample.parameters","contentType":"file"}],"templateDirectorySuggestionUrl":null,"readme":{"displayName":"README.txt","richText":"
/*!\n\n\\mainpage Visual and Interactive Test Automation for SDL 2.0\n\n\\section license_sec License\nCheck the file \\c COPYING.txt for licensing information.\n\n\\section intro_sec Introduction\nThe goal of this GSoC project is to automate the testing of testsprite2.\ntestsprite2 takes 26 parameters which have thousands of valid combinations and is\nused to validate SDL's window, mouse and rendering behaviour. By having a test\nharness that runs testsprite2 with various command line argument strings and\nvalidates the output for each run, we can make testing an easier task for\nmaintainers, contributors and testers. The test harness can be used by a continuous\nintegration system (like buildbot or jenkins) to validate SDL after checkins.\n\nSDL Homepage: http://libsdl.org/\n\n\\section build_sec Building\n\n\\subsection build_linux Building on Linux/Cygwin\n<tt>./autogen.sh; ./configure; make;</tt>\n\n\\subsection build_windows Building on Windows\nUse the Visual Studio solution under \\c SDL/VisualC/visualtest.\n\n\\section docs_sec Documentation\nDocumentation is available via Doxygen. To build the documentation, cd to the\nSDL/visualtest/docs directory and run \\c doxygen. A good starting point for\nexploring the documentation is \\c SDL/visualtest/docs/html/index.html\n\n\\section usage_sec Usage\nTo see all the options supported by the test harness, just run \\c testharness\nwith no arguments.\n\nAt the moment the following options are supported:\n\\li \\c sutapp - Path to the system under test (SUT) application\n\\li \\c sutargs - Launch the SUT with the specified arguments string\n\\li \\c timeout - The maximum time after which the SUT process will be killed;\n\tpassed as hh:mm:ss; default 00:01:00\n\\li \\c variator - Which variator to use; see \\ref variators_sec\n\\li \\c num-variations - The number of variations to run for; taken to be \n\t1 for the random variator and ALL for the exhaustive variator by default\n\\li \\c no-launch - Just print the arguments string for each variation without\n\tlaunching the SUT or performing any actions\n\\li \\c parameter-config - A config file that describes the command line parameters\n\tsupported by the SUT; see \\ref paramconfig_sec or the sample *.parameters files\n\tfor more details\n\\li \\c action-config - A config file with a list of actions to be performed while\n\tthe SUT is running; see \\ref actionconfig_sec or the sample *.actions files\n\\li \\c output-dir - Path to the directory where screenshots should be saved; is\n\tcreated if it doesn't exist; taken to be \"./output\" by default\n\\li \\c verify-dir - Path to the directory with the verification images; taken to \n\tbe \"./verify\" by default\n\nPaths can be relative or absolute.\n\nAlternatively, the options can be passed as a config file for convenience:\n\n<tt>testharness \\-\\-config testsprite2_sample.config</tt>\n\nFor a sample, take a look at the *.config files in this repository.\n\nWe can also pass a config file and override certain options as necessary:\n<tt>testharness \\-\\-config testsprite2_sample.config \\-\\-num-variations 10</tt>\n\nNote: You may find it convenient to copy the SUT executable along with any\nresources to the test harness directory. Also note that testsprite2 and its\nresources (icon.bmp) are automatically copied when using the Visual Studio\nsolution.\n\n\\subsection usageexamples_subsec Usage examples:\n\nPassing a custom arguments string:\n<tt>testharness \\-\\-sutapp testsprite2 \\-\\-sutargs \"\\-\\-cyclecolor \\-\\-blend mod\n\\-\\-iterations 2\" \\-\\-action-config xyz.actions</tt>\n\nUsing the random variator:\n<tt>testharness \\-\\-sutapp testsprite2 \\-\\-variator random \\-\\-num-variations 5\n\\-\\-parameter-config xyz.parameters \\-\\-action-config xyz.actions</tt>\n\n\\subsection config_subsec Config Files\nConfig files are an alternate way to pass parameters to the test harness. We\ndescribe the paramters in a config file and pass that to the test harness using\nthe \\-\\-config option. The config file consists of lines of the form \"x=y\" where\nx is an option and y is it's value. For boolean options, we simply give the name\nof the option to indicate that it is to be passed to the testharness.\n\nThe hash '#' character can be used to start a comment from that point to the end\nof the line.\n\n\\section paramconfig_sec The SUT Parameters File\nTo generate variations we need to describe the parameters the will be passed to\nthe SUT. This description is given in a parameters file. Each line of the parameters\nfile (except the blank lines) represents one command line option with five\ncomma separated fields:\n<tt>name, type, values, required, categories</tt>\n\n\\li \\c name is the name of the option, e.g., \\c \\-\\-cyclecolor.\n\\li \\c type can have one of three values - integer, boolean and enum.\n\\li \\c values - for integer options this is the valid range of values the option\n\tcan take, i.e., [min max]. For enum options this is a list of strings that\n\tthe option can take, e.g., [val1 val2 val3]. For boolean options this field\n\tis ignored.\n\\li \\c required - true if the option is required, false otherwise.\n\\li \\c categories - a list of categories that the option belongs to. For example,\n\t[video mouse audio]\n\nJust like with config files, hash characters can be used to start comments.\n\n\\subsection additionalnotes_subsec Additional Notes\n\n\\li If you want to have an option that always takes a certain value, use an enum\n\twith only one value.\n\\li Currently there isn't any way to turn an option off, i.e., all options will\n\tbe included in the command line options string that is generated using the\n\tconfig. If you don't want an option to be passed to the SUT, remove it from\n\tthe config file or comment it out.\n\n\\section variators_sec Variators\nVariators are the mechanism by which we generate strings of command line arguments\nto test the SUT with. A variator is quite simply an iterator that iterates through\ndifferent variations of command line options. There are two variators supported at\nthe moment:\n\\li \\b Exhaustive - Generate all possible combinations of command line arguments\n\tthat are valid.\n\\li \\b Random - Generate a random variation each time the variator is called.\n\nAs an example, let's try a simple .parameters file:\\n\n<tt>\n\\-\\-blend, enum, [add mod], false, [] \\n\n\\-\\-fullscreen, boolean, [], false, []\n</tt>\n\nThe exhaustive variator would generate the following four variations:\\n\n<tt>\n\\-\\-blend add \\n\n\\-\\-blend mod \\n\n\\-\\-blend add \\-\\-fullscreen \\n\n\\-\\-blend mod \\-\\-fullscreen \\n\n</tt>\n\nThe random variator would simply generate a random variation like the following:\\n\n<tt>\\-\\-blend mod</tt>\n\n\\section actionconfig_sec The Actions File\nOnce the SUT process has been launched, automated testing happens using a mechanism\ncalled actions. A list of actions is read from a file and each action is performed\non the SUT process sequentially. Each line in the actions file describes an action.\nThe format for an action is <tt>hh:mm:ss ACTION_NAME additional parameters</tt>.\nThere are five actions supported at the moment:\n\\li \\b SCREENSHOT - Takes a screenshot of each window owned by the SUT process. The\n\timages are saved as \\c [hash]_[i].bmp where \\c [hash] is the 32 character long\n\thexadecimal MD5 hash of the arguments string that was passed to the SUT while\n\tlaunching it and \\c i is the window number. i = 1 is an exceptional case\n\twhere the \\c _[i] is dropped and the filename is simply \\c [hash].bmp\\n\n\tNote: The screenshots are only of the window's client area.\n\\li \\b VERIFY - Verifies the screenshots taken by the last SCREENSHOT action by\n\tcomparing them against a verification image. Each \\c [hash]_i.bmp image output\n\tby the SCREENSHOT action is compared against a \\c [hash].bmp image in the\n\tverify-dir.\n\\li \\b QUIT - Gracefully quits the SUT process. On Windows this means sending a\n\tWM_CLOSE message to each window owned by the SUT process. On Linux it means\n\tsending a SIGQUIT signal to the SUT process.\n\\li \\b KILL - Forcefully kills the SUT process. This is useful when the SUT process\n\tdoesn't respond to the QUIT action.\n\\li <b>LAUNCH [/path/to/executable] [args]</b> - Runs an executable with \\c [args]\n\tas the arguments string.\n\nJust like with config files, hash characters can be used to start comments.\n\n\\section contint_sec Continuous Integration (CI)\nOne of the goals of the project was to create a test harness that integrates\nwith CI systems to provide automated visual and interactive testing to SDL.\n\nAt the moment the test harness can be run in two modes that are useful for CI:\n\\li Crash testing mode - launch the SUT with every variation and all parameters,\n\treport to the CI if there's a crash\n\\li Visual testing mode - launch and visually verify the SUT for a smaller subset\n\tof the parameters\n\nLook at the launch_harness.sh/launch_harness.cmd for an example scripts that run the\ntest harness for all variations with all parameters and report an error on a crash.\nThe script uses the testsprite2_crashtest config, so remember to copy those files\nover to the test harness executable directory along with the script.\n\n\\section todo_sec TODOs\n\\li Allow specifying a clipping box along with the VERIFY action, i.e., hh:mm:ss\n\tVERIFY x, y, w, h\n\\li Add support for spaces between the equals sign in test harness config files\n\\li Implement the SCREENSHOT action on Linux\n\\li Add a pairwise variator\n\\li Add actions to inject keyboard/mouse events\n\\li Add actions to manipulate the SUT window, e.g., minimize, restore, resize\n\\li Add support to load and save screenshots as .pngs instead of .bmps\n\n\\section issues_sec Known Issues\n\\li The QUIT action does not work on a testsprite2 process with multiple windows.\n\tThis appears to be an issue with testsprite2.\n\\li The SCREENSHOT action doesn't capture the testsprite2 window correctly if the\n\t--fullscreen option is supplied. It works with --fullscreen-desktop, however.\n\n\\section moreinfo_sec More Information\n\nAuthor Contact Info:\\n\nApoorv Upreti \\c \\<apoorvupreti@gmail.com\\>\n\nOther useful links:\n- Project Repository: https://bitbucket.org/nerdap/sdlvisualtest\n- Project Wiki: https://github.com/nerdap/autotestsprite2/wiki\n- Project Blog: http://nerdap.github.io\n- Verification images for testsprite2_blendmodes: https://www.dropbox.com/s/nm02aem76m812ng/testsprite2_blendmodes.zip\n- Verification images for testsprite2_geometry: https://www.dropbox.com/s/csypwryopaslpaf/testsprite2_geometry.zip\n*/\n