Java test framework - testng in detail (part 1)

I recall back to 2013, when I first got myself into automation test with Java language then testng is really de-facto framework. No magic about this choice since I worked for an out-sourcing company and the most effective way for such company to get stuffs done is: copy and paste. Once they receive request from clients to develop automation test, they just copy the most recent test repository then they do:

• rename package to hide origin of the work
• remove all tests from the existing repository
• add some dummy tests as a sign of readiness

It worked pretty well since most clients, frankly, they don’t give a shit about test code as long as it can run, it can verify stuffs, and it could save them ton of money in long run. At least, this saving factor is quite important when the client has financial target to cut from development spending.

Typically, newbie first test from first onboarding week looks like this:

@Test
public class TestSomeThing {
    void testArea() {
        // some code test
    }

    void testPoint() {
        // some other testCode
    }

    @BeforeMethod
    void prepare() {
        // setup code
    }

    @AfterMethod
    void teardown() {
        // clean up resource
    }
}

Then by simply clicking a few steps to ask IDE run tests for us. This can be done either by Eclipse IDE or IntelliJ IDE. In 2013, Eclipse IDE was still “something” in Java IDE land. In fact, most of my teammates use it for as daily driver IDE. Lateral, we all move to IntelliJ for superior experience.

At that time, I hadn’t asked why do the test run but follow instruction from previous member of the team. It works really smooth for me until I need more customization for the work I delivered to the project:

• I want my tests can run parallel when they are not conflict to each other by a careful test data design or randomization
• I want crystal logging from a test even the test spreads across multiple java

This strikes me quite heavily since I don’t have coding background and all I got till that moment is following told instruction. That the root of all my curiosity for coding till now: I really want to understand the magic behind clicking some buttons then test code run. The journey started there, and now I look back, I really want to share my knowledge around my very first question:

How does testng run a test from a particular java class file?

Short answer: It was a combination of tools mixed together to make test execution happen. In my first projects, they are:

- maven build tool
- maven surefire plugin
- testng framework
- my java class which have test code

Figure: A sample peudo Java test class

Long answer: A lot of thing happens during test

I would like to skip first three steps this blog post to focus on testng at step 4 to describe the magic of testng. As we know, for Java language, when we have a class, and we want to invoke its functionality by calling its methods we need to have an instantiated object of the class then we explicitly write code like following entry point:

public class ManualRun {
    public static void main(String[] args) {
        TestSomeThing testSomeThing = new TestSomeThing();
        testSomeThing.testArea();
        testSOmeThing.testPoint();
    }
}

The interesting thing is that we cannot find such code in our example not to mention, we don’t write any code to handle execution and collect metadata for execution. This is exactly what testng, a test framework, does for us:

When surefire plugin kicks testng with passed configuration, testng will:

1. testng constructs objects to hold metadata on how to run a suite (can have many suites)
2. testng instantiate its suite runner object to trigger suite execution
3. testng suite runner instantiates test runner object to run a test
4. testng runner instantiates objects which hold test code on test methods (via Reflection API) - note on sample ManualRun.java we need to specifically create a new object by calling new keyword but using Reflection API, we can instantiate java object with default constructor.
5. testng runner invokes test methods from the object (this is where our test code happens)
6. testng keeps track of execution metadata such as:
   • test result
   • test duration
   • exception
   • logging
7. Repeat test runner and suite runner for the rest of passed configuration from surefire
8. Stop test JVM when last test is executed.

Figure: Testng from its entry point.

So, that’s it. Next time you run a test by clicking Play button next to Java class from IntelliJ, now you can understand why, when, which executes your test code.