|
|
|
|
|
Advanced test techniques |
|
Integration with TestDirector |
|
Customising EMOS Framework |
|
Troubleshooting |
|
|
|
|
Re-using test data |
|
Managing complex tests |
|
Increasing test robustness |
|
Invoking dummy-mode |
|
Invoking retry-mode |
|
|
|
|
|
Test sequences that are often repeated in
various tests can be easily reused via the LINK mechanism. |
|
|
|
Recommended procedure: |
|
|
|
identify test sequences that can be re-used |
|
allocate new test table (optional) |
|
re-organise the test data into new tests |
|
assign meaningful names to relocated test data |
|
use LINK command from other tests to reuse the
relocated data when you need it |
|
|
|
|
|
|
|
|
Re-using test data |
|
Managing complex tests |
|
Increasing test robustness |
|
Invoking dummy-mode |
|
Invoking retry-mode |
|
|
|
|
|
If your tests grow big so that numerous columns
and/or different test tables are required to complete a single test, then
create an top-level test that only contains LINK instructions. |
|
Recommended procedure: |
|
|
|
create an intermediary test table |
|
the test sequences only contain
LINK-instructions |
|
provide the description to each test case |
|
place this test table in a separate directory
together with all linked tables and call the table “Tests.xls” |
|
you may create your own variation of ¸ however make sure
to be consistent with names of such tables |
|
|
|
|
|
|
|
|
Re-using test data |
|
Managing complex tests |
|
Increasing test robustness |
|
Invoking dummy-mode |
|
Invoking retry-mode |
|
|
|
|
|
Test suites often need to run unattended for
hours. You must apply programming techniques to make them robust (i.e. able
to recover from failures). |
|
|
|
Recommended procedure: |
|
create the test block named “Prologue” (or the
like) |
|
this block must be able to set the AUT to a
particular initial state no matter what state the AUT was in before |
|
this block may require parameters, provide them |
|
start every test case with test block “Prologue” |
|
the rest of the test cases may assume the
initial state |
|
you may need a few such blocks for different
initial states |
|
apply the “re-using test data” technique to
reduce the amount of data in test tables |
|
|
|
|
For the purpose of batch testing the test suite
must be capable of handling unexpected behaviour of the tested application. |
|
Individual test should be capable of eventually
re-synchronising themselves with the application so that only small parts
of the test suite fail due to errors. |
|
Typical doctrine to handle such situation is to
make each test “responsible” for setting the initial state before the
actual test execution and to perform the “cleanup” after the test
execution. |
|
This rule is not sufficient due to the fact that
in case of an error a test may not be able to perform the desired cleanup.
In other words it is the initialising routine who is responsible for the
cleanup if nothing else works. |
|
|
|
|
|
|
In EMOS Framework we tend to use the test block
named “Prolog” to perform the robust initialising. |
|
Its purpose is to establish a certain initial
state of the application regardless of the application’s state. In other
words if the application has been found in the expected state, Prolog does
nothing. Otherwise it attempts to establish the desired state by for
example closing open windows, re-invoking the application, etc. |
|
Prolog must handle the login process if the
tested application requires users to log in. |
|
Once prolog-logic has been established, each
test case MUST start with the Prolog block. |
|
|
|
|
|
|
|
|
Re-using test data |
|
Managing complex tests |
|
Increasing test robustness |
|
Invoking dummy-mode |
|
Invoking retry-mode |
|
|
|
|
|
In “dummy” mode EMOS Framework executes the test
suite up to the point of invoking the test block but not further. |
|
In other words the test is run but the test
primitives are not performed - the test suite only “pretends” to be running
but nothing actually gets invoked. |
|
Dummy mode serves two main purposes: |
|
it generate the test documentation |
|
it can be used to measure the “overhead” of
reading the test data from the date tables. |
|
To invoke dummy-mode call:
FRM_STP_set_dummy_step_mode(
TRUE ); |
|
To return from dummy-mode call:
FRM_STP_set_dummy_step_mode(
FALSE ); |
|
|
|
|
Re-using test data |
|
Managing complex tests |
|
Increasing test robustness |
|
Invoking dummy-mode |
|
Invoking retry-mode |
|
|
|
|
In “retry” mode EMOS Framework records the state
of executed tests of a particular test suite. |
|
In case of failure and re-execution of the same
test suite a dialog box is displayed (only in interactive mode) and the
user is asked whether the failed run should be continued (from the point of
failure) or start from the beginning. |
|
The purpose of the retry-mode is to simplify the
process of re-execution of test runs and speed up the test
development/debugging/analysis process. |
|
To invoke retry-mode call:
FRM_DRV_enable_retry(); |
|
To return from retry-mode call:
FRM_DRV_disable_retry(); |
|
|
|
|
|
|
|
The logic for “retry” mode is not finished yet.
It is still buggy in some places but useful in some situations. We plan to
finalise it |
|
Since EMOS Framework is open source this is a
good place to try your programming skills. Our wish list starts with: |
|
fix bugs |
|
improve the user interface (e.g. re-starting at
the point of failure is sometimes not what is needed, due to the nature of
failure it is sometimes necessary to re-start at some earlier point, i.e.
offer surrounding test blocks) |
|
refine the granularity of re-starting point
(e.g. re-starting at the test primitive level versus re-starting at test
block level) |
|
|
|
|
Advanced test techniques |
|
Integration with TestDirector |
|
Customising EMOS Framework |
|
Troubleshooting |
|
|
|
|
Test suites can be completely replaced by
TestDirector test sets |
|
|
|
Rule 1
Test drivers and test libraries created with EMOS technique should
not physically reside in TestDirector repository
(i.e. they must also work without TD!) |
|
|
|
Rule 2
TestDirestor tests that call EMOS
Framework tests may reside in TestDirector repository |
|
|
|
Rule 3
startup script is
still required (it may be called automatically by WR or as the first script
in TD test set) |
|
|
|
|
|
|
|
|
|
|
Advanced test techniques |
|
Integration with TestDirector |
|
Customising EMOS Framework |
|
Troubleshooting |
|
|
|
|
The main part of EMOS Framework can be found in
library emos_frm_gui_lib. |
|
|
|
This library contains the so-called FRM-wrappers
for standard WinRunner functions such as edit_set(), list_select_item(),
etc. In order to accommodate new features you will need to extend these
features. |
|
|
|
Note:
Do not modify the original library because you
will not be able to
install updates of EMOS
Framework without
running into troubles! |
|
|
|
|
|
|
Recommended procedure: |
|
|
|
do not modify the original library! |
|
create new library, recommendation:
LIB/aut_frm_gui_lib |
|
copy one family of functions from the original
library, for example: |
|
FRM_edit_set() |
|
FRM_SET_ edit_set() |
|
FRM_CHK_ edit_set() |
|
FRM_ATR_ edit_set() |
|
FRM_GEN_ edit_set() |
|
rename and modify your functions |
|
load your library with startup test |
|
|
|
|
Advanced test techniques |
|
Integration with TestDirector |
|
Customising EMOS Framework |
|
Troubleshooting |
|
|
|
|
|
If you call an invalid suite table, an error is
reported in the test results log. |
|
|
|
Scenario: |
|
call wrong test suite from the kickoff |
|
this block must be able to set the AUT to a
particular |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Notizen