- tl;dr
- BDD connects tech and business
- Cucumber currently has no
@BeforeAll
or@AfterAll
- Train yourself a hamster
- No standard solution in Cucumber JVM
- Other frameworks’ solutions
- Looking for workarounds in Cucumber
- Injecting results
- Tapping into Cucumber’s lifecycle
- Summary
tl;dr
Cucumber JVM does not feature a @BeforeAll
or @AfterAll
lifecycle hook to run setup and teardown procedures. This article shows how to achieve global setup and teardown with
- either plain Cucumber JVM
- or Cucumber JVM combined with a JUnit wrapper
- or from the outside using Apache Maven.
All variants are demonstrated by working code examples on github.
BDD connects tech and business
At metamorphant we often face the challenge of bringing tech and business closer together. One method from our toolbox are BDD-style feature tests. A popular approach is to phrase tests in Gherkin – a human-readable, domain-specific language for scenario testing. Users of Java or other JVM languages will typically choose either JBehave or Cucumber for that purpose.
Cucumber currently has no @BeforeAll
or @AfterAll
When testing complex scenarios, sometimes you need to setup an expensive resource before starting the first test and tear it down after the last test.
Example: Test Database
One of our customers had a dependency on a MongoDB NoSQL Database. For tests they wanted to adopt flapdoodle embedded MongoDB. In general, non-Java DBs will run as a separate OS process. This rule equally applies when packaging the component in a Docker container. Startup and teardown take at least some seconds. So, you do not want to afford it for each test. Sharing a DB instance for multiple tests is feasible, if you have a way of cleanly resetting the DB before the individual tests.
Train yourself a hamster
For the purpose of this article I will switch to a simpler scenario. My little test scenario revolves around hamster training (thanks for the inspiration, Michael Keeling). It features the bare minimum of a BDD test.
Like setting up an embedded DB, the training of a hamster takes a long time and is expensive in terms of resources. I will simulate that in the hamster code by a simple Thread.sleep(5000)
.
No standard solution in Cucumber JVM
It turns out, that this kind of setup/teardown scenario is currenty not well supported by Cucumber JVM. Cucumber JVM brings its own JUnit Runner and controls its internal lifecycle independently. Users define the details like glue code Java packages and feature file paths by annotation of the test class. It is not unusual to end up with an empty test class:
The execution is controlled mostly by the step implementations. There is a fixed set of supported steps:
@Given
,@When
,@Then
steps are executed when used in a scenario@Before
is executed before a scenario@After
is executed before a scenario
There is no equivalent of @BeforeAll
and @AfterAll
. No step runs before the whole feature or even before the whole test suite. The corresponding github issue 515 about adding @BeforeAll and @AfterAll hooks is highly popular. It obviously struck a nerve. But, despite the high interest, it is unclear when – if ever – the Cucumber upstream project will release such new features.
Hence, my customer asked for help in implementing a workaround with current Cucumber versions. Issue 515 already describes some workarounds but lacks complete, workable examples.
Other frameworks’ solutions
How is the same problem solved in other test frameworks?
I will compare the solutions of the most common Java test toolkits:
I do not have a clue about other frameworks like Concordion or FitNesse. For a library like jgiven the answer is simple: it piggybacks on other test frameworks like JUnit 4 as an embedded fluid DSL and inherits their test structure and lifecycle.
Standard solution in plain JUnit
Plain JUnit 4 structures tests by methods and classes. Logically, the hook annotations are
@BeforeClass
and@AfterClass
for wrapping a test class’s lifecycle and@Before
and@After
for wrapping each test method.
Resp. for JUnit 5 the annotations have been renamed to
@BeforeAll
and@AfterAll
for test classes and@BeforeEach
and@AfterEach
for test methods.
I will use JUnit annotations for one of our proposed workarounds later.
Standard solution in Spock
The awesome Spock Framework follows a similar approach as JUnit and uses magic method names
setupSpec()
andcleanupSpec()
for test classes andsetup()
andcleanup()
for test methods.
As a special gimmick you can extend base test classes and chain setup and cleanup methods superclass-first.
Standard solution in JBehave
JBehave uses a slightly different lingo than cucumber to structure the test hierarchy:
- A JBehave scenario comprises multiple JBehave steps. It is equivalent to a Cucumber scenario.
- A JBehave story comprises multiple JBehave scenarios. It is equivalent to a Cucumber feature.
- Multiple JBehave stories can be combined to a collection of stories.
JBehave’s lifecycle concept provides detailed hook-in points to attach logic to each element. You can either call steps from your story files using a special syntax or programmatically using JBehave’s annotations.
@BeforeScenario
and@AfterScenario
wrap a scenario.@BeforeStory
and@AfterStory
wrap a single story@BeforeStories
and@AfterStories
wrap a collection of stories.
Looking for workarounds in Cucumber
I will demonstrate the workarounds using the hamster scenarios described above. The hamster implementation in class de.metamorphant.blog.hamster.Hamster
is the same for all 4 examples.
All examples except the maven failsafe approach contain a class de.metamorphant.blog.hamster.HamsterUtil
in addition. It provides the reusable hamster training logic:
The challenges to be solved for our tests are 2-fold:
- How can I tap into the test execution lifecycle to run setup before all and teardown after all Cucumber tests?
- How can I inject results from the setup (e.g. DB connection data, random ports, …) into the object under test?
Injecting results
Injection becomes necessary, whenever the hooks are separate from the step definition class. Possible injection strategies are:
- as configuration e.g. by JVM system properties, environment variables, files, …
- through static variables and static methods
The Cucumber JVM documentation warns about the use of static variables and recommends using other state management mechanisms. In the case of emulating @BeforeAll
/@AfterAll
, the static behaviour is exactly what we need and the warning does not apply.
Tapping into Cucumber’s lifecycle
In a typical Maven project, Cucumber runs in a specific context:
- Maven performs whatever is configured until the relevant test phase (either
test
orintegration-test
) - Maven starts the JUnit test (by default in a new JVM, see Maven Fork option documentation)
- JUnit calls its
@BeforeClass
hooks - JUnit delegates to the Cucumber Runner
- Cucumber executes all scenarios from all features. During the feature execution, Cucumber reports about each internal lifecycle transition by events. For each scenario Cucumber calls (in that order):
- Before hooks
- Background steps
- Scenario steps
- After hooks
- Cucumber finishes execution
- JUnit calls its
@AfterClass
hooks - Maven performs whatever is configured after the test phase
You probably already noticed the available hook-in-points. I will demonstrate them one by one. All examples use Cucumber’s Java 8 flavour.
Use a Before and a shutdown hook
The infamous issue 515 starts with a recommendation to
- use a Before step,
- check if initialization already happened,
- lazily initialize if not and
- register a JVM Runtime shutdown hook to perform the shutdown.
The global hook solution in my examples looks like this:
Looks kinda clean, doesn’t it? … Buzzinga! It doesn’t.
You can see the true lifecycle in the test’s console output:
Watch out for the lines
- Cucumber Before hook called; starting to train a hamster
- JVM shutdown hook called; gracefully shutting down hamster
When running with Maven this works out fine, as the Maven Surefire Plugin forks a separate JVM for the test execution. If you e.g. run your tests in an IDE inside a reused, long-lived JVM process, your teardown will not be called after test completion.
⊕⊕⊕ Pros ⊕⊕⊕:
- Works for most use cases
⊖⊖⊖ Cons ⊖⊖⊖:
- Dirty trick:
java.lang.Runtime.getRuntime().addShutdownHook(...)
is a JVM feature and in no way related to Cucumber. - Asymmetric: Setup and Teardown hook into different layers
- Teardown will only happen on JVM termination
- No per-feature lifecycle possible
Variant: Use a Feature Background
You might think, that the Hamster training is business relevant in our scenarios. In that case, you can express the setup more explicitly in a Feature’s Background steps. Lifecycle-wise it is mostly equivalent to the Before hook and I will skip a separate demo here.
Implement a Cucumber EventListener
As a next variant, I want to explore Cucumber’s Lifecycle Event notifications. For that purpose, I will create and register a plugin PortSetupLifecycleHandler
of type EventListener
.
I register it by annotating the test class:
The output shows the result:
The setup and teardown are bound to the event types TestRunStarted
and TestRunFinished
now. Both are under the control of Cucumber. No outside party is involved. This will also work when using a different driver, e.g. Cli.main
or an IDE’s custom implementation.
Unfortunately, there are no event types for binding by feature. You either choose the coarse grained event type for the whole test run or you choose the fine grained event type for an individual scenario. The middle ground is not covered.
⊕⊕⊕ Pros ⊕⊕⊕:
- Cleanly uses Cucumber’s lifecycle events
- Works even with non-JUnit drivers like
Cli.main
⊜⊜⊜ Neutral ⊜⊜⊜:
- Test infrastructure tightly coupled to Cucumber
⊖⊖⊖ Cons ⊖⊖⊖:
- No per-feature lifecycle possible
Use JUnit’s lifecycle to wrap Cucumber’s
When running with JUnit only, you can leverage JUnit’s @BeforeClass
and @AfterClass
annotations. Additionally, my JUnit wrapper example includes @Before
and @After
annotations: They will not be called, as after class initialization Cucumber’s test runner takes over.
One typical problem with the JUnit-only assumption are IDEs. For example, IntelliJ’s Cucumber plugin uses the Cucumber Command Line Runner Cli.main
and will not run the tests with a JUnit driver. If you go that path, you are in for the ‘works on my machine’ effect.
The output of this variant is:
⊕⊕⊕ Pros ⊕⊕⊕:
- Cleanly wraps the Cucumber test
- Easy to read
- Easy to write
- Easy to reason about
⊖⊖⊖ Cons ⊖⊖⊖:
- No per-feature lifecycle possible
- Depends on JUnit runner
- implies: breaks IntelliJ IDE usage resp. demands additional setup
Leverage Maven lifecycle’s integration test features
A totally different approach is to acknowledge the tests’ nature as integration tests and set up all dependencies completely from the outside. For Maven projects this is done by using the Maven Failsafe Plugin instead of the Maven Surefire Plugin. Typically, this means configuring the Maven Build Lifecycle phases
pre-integration-test
,integration-test
,post-integration-test
,
which are all covered by the higher-level verify
phase. The pom.xml of my example project shows an example configuration. It uses the Maven AntRun Plugin as a simple way to produce effects and the Build Helper Maven Plugin to generate random ports.
The heavy lifting is now moved from the Java test code into the Maven configuration. The injection of parameters into the test runtime is done using system properties. Notice that all inputs are generated beforehand and passed by property name convention. Even the pre-integration-test
phase takes the random hamster.port
as input from outside.
This approach works smoothly, but only as long as you find proper Maven plugins to achieve your setup and teardown goals.
mvn verify
produces:
As a side effect, you can now run your test in other environments in exactly the same way, e.g. after deploy to a Quality Assurance stage. Just create a different build profile that provides the necessary system properties from a different source.
⊕⊕⊕ Pros ⊕⊕⊕:
- Clean separation between setup / teardown and test logic
- Battle-proven pattern
- Test implementation technology agnostic
- Can be used to test against live environments without changing the test code
⊖⊖⊖ Cons ⊖⊖⊖:
- Only feasible, if Maven plugins for setup / teardown are available (or you are willing to create them)
- No per-feature lifecycle possible
Summary
You have seen 4 ways to work around the missing solution for issue 515. None of them enables per-feature lifecycle hooks. However, all of them enable per-test-run setup and teardown.
Which one you choose depends on your circumstances.
Post header background image by S. Hermann & F. Richter from Pixabay.