Although the programming of automated tests should contribute to the overall test effort, it usually does not make up
the entire test effort. In fact, test environments that are based on a complete automation approach end up spending
more time on test automation than on testing. Before you begin developing automated test scripts, consider first
whether it is more efficient to perform manual testing. Some aspects of an application are more efficiently tested
manually (for example, GUI testing versus data-drive testing). If you decide to program automated test scripts, examine
what aspects of your test scripting can be automated and begin designing your scripts.
Design your automated tests
Without some level of design of your automated tests, introducing automation into your testing effort can lead to more
problems than it solves. You should consider developing your automated tests according to a lifecycle with automation
test requirements, design, testing of the automation tests, and implementation of the automation tests. This approach
can be informal or formal depending on your project needs. By designing the programming of your automated tests, you
can avoid spending time programming the wrong tests, re-working programmed tests, deciphering different coding styles
in the programming of the tests, etc.
Recorded versus programmed scripts
Although there are clear benefits to recorded scripts (for example, ease of creation or ability for novice testers to
learn a scripting language), recorded scripts also present their own problems. The disadvantages of playback scripts
are well known. They are deceptively easy to create but very difficult to update. Problems with script reliability,
hard-coded data values, or changes to the application under test and the need to re-record are well-documented. On the
other hand, programming scripts can present difficulties of their own: they are difficult for the novice tester to
create, they can require substantial time and effort to develop, and they can be difficult to debug. Most test tooling
makes these issues less problematic by providing the tester script support functions, such as ways to establish target
of test lists, systematic ways to program verification point, point to datapools, build commands into the script (for
example, sleeper commands), comment the script, and document the script. Another major advantage, which is often
overlooked, of using testing tooling to mitigate these risks is the ability to add to an existing script in the form of
making corrections to an existing script, testing new features of a test target or application under test, or resuming
a recording after an interruption.
Functional and performance test scripts
When discussing automating test scripts, it is important to distinguish between functional and performance tests. Most
discussions of programming automated test scripts focus on testing the functionality of an application. This is not
inappropriate, since a lot of automated testing focuses on functional testing. Performance test scripting, however, has
its unique characteristics. Performance test automation provides you with the ability to programmatically set workloads
by adding user groups to test loads under group usage, setting think time behavior, running tests randomly or at set
rates, or setting the duration of a run. Performance test automation also allows you to create and maintain schedules
for your tests.
Testing test scripts
When testing your test scripts, keep in mind whether you are testing recorded or programmed test scripts. For recorded
scripts, much of the debugging of the script consists of errors that are introduced due to changes in the test target
or test environment. When you run a recorded test script, consider the test target of the script. Some test automation
tools capture this information as a part of the test script. Debugging a recorded script consists largely of
determining whether changes in the target have created error conditions in the script. In general, there are two main
categories to examine here: changes in the UI and test session sensitive data (for example, date stamped data). In most
cases, discrepancies between recording and playback cause errors in your recorded test scripts.
Testing programmed test scripts involves many of the same debugging techniques you would apply to debugging an
application. Consider both the flow control logic and the data aspects of your script. Automated testing tools provide
you with test script debugging IDEs as well as datapool management features that facilitate this type of testing.
During execution of test scripts, a test that uses a datapool can replace values in the programmed test with variable
test data that is stored in the datapool.