BDD with Cucumber vs Spock

In this post I compare Cucumber with Spock for BDD. To do this I consider two simple test cases of

• a book library that offers a 'search by publication year' feature, and
• a salary manager that modifies the salary of one or more employees. 

I take the cucumber code from this bitbucket repository. The Spock code can be found in the spock-example repository. The source code (domain model) is the same in both cases. Only the test code changes.

The Book Library Test Case

With Cucumber

Cucumber offers a DSL for writing feature specifications in English, called Gherkin, that is then associated with test code. A feature contains one or more scenarios. Each scenario consists of steps. A simple feature with one scenario for a book library could look like:

	Feature: Book search
	To allow a customer to find his favourite books quickly, the library must offer multiple ways to search for a book.
	Scenario: Search books by publication year
	Given a book with the title 'One good book', written by 'Anonymous', published in 14 March 2013
	And another book with the title 'Some other book', written by 'Tim Tomson', published in 23 August 2014
	And another book with the title 'How to cook a dino', written by 'Fred Flintstone', published in 01 January 2012
	When the customer searches for books published between 2013 and 2014
	Then 2 books should have been found
	And Book 1 should have the title 'Some other book'
	And Book 2 should have the title 'One good book'

view raw search_book.feature hosted with ❤ by GitHub

Feature, Scenario, Given, When, Then, And are cucumber reserved keywords, while the rest is English. Each sentence starting with Given, When, Then defines a step. If you run this test, cucumber will search for step method implementations associated with the steps defined in Gherkin. The association is done with annotations containing proper regular expressions. If no such implementation exists Gherkin will print in the console the signature of the methods you have to implement, like:

	You can implement missing steps with the snippets below:
	@Given("^a book with the title 'One good book', written by 'Anonymous', published in (\\d+) March (\\d+)$")
	public void a_book_with_the_title_One_good_book_written_by_Anonymous_published_in_March(int arg1, int arg2) throws Throwable {
	// Write code here that turns the phrase above into concrete actions
	throw new PendingException();
	}
	@Given("^another book with the title 'Some other book', written by 'Tim Tomson', published in (\\d+) August (\\d+)$")
	public void another_book_with_the_title_Some_other_book_written_by_Tim_Tomson_published_in_August(int arg1, int arg2) throws Throwable {
	// Write code here that turns the phrase above into concrete actions
	throw new PendingException();
	}
	@Given("^another book with the title 'How to cook a dino', written by 'Fred Flintstone', published in (\\d+) January (\\d+)$")
	public void another_book_with_the_title_How_to_cook_a_dino_written_by_Fred_Flintstone_published_in_January(int arg1, int arg2) throws Throwable {
	// Write code here that turns the phrase above into concrete actions
	throw new PendingException();
	}
	@When("^the customer searches for books published between (\\d+) and (\\d+)$")
	public void the_customer_searches_for_books_published_between_and(int arg1, int arg2) throws Throwable {
	// Write code here that turns the phrase above into concrete actions
	throw new PendingException();
	}
	@Then("^(\\d+) books should have been found$")
	public void books_should_have_been_found(int arg1) throws Throwable {
	// Write code here that turns the phrase above into concrete actions
	throw new PendingException();
	}
	@Then("^Book (\\d+) should have the title 'Some other book'$")
	public void book_should_have_the_title_Some_other_book(int arg1) throws Throwable {
	// Write code here that turns the phrase above into concrete actions
	throw new PendingException();
	}
	@Then("^Book (\\d+) should have the title 'One good book'$")
	public void book_should_have_the_title_One_good_book(int arg1) throws Throwable {
	// Write code here that turns the phrase above into concrete actions
	throw new PendingException();
	}

view raw cucumber_output.java hosted with ❤ by GitHub

You need to copy paste this code into a test class and then provide implementations for each of them. Notice however that some methods can be merged with each other if we provide smarter regular expressions. For instance, the three methods annotated with @Given can be merged into one by providing regular expressions for the title, author and publishing date. The most concise implementation is:

	public class BookSearchSteps {
	Library library = new Library();
	List<Book> result = new ArrayList<>();
	@Given(".+book with the title '(.+)', written by '(.+)', published in (.+)")
	public void addNewBook(final String title, final String author, @Format("dd MMMMM yyyy") final Date published) {
	Book book = new Book(title, author, published);
	library.addBook(book);
	}
	@When("^the customer searches for books published between (\\d+) and (\\d+)$")
	public void setSearchParameters(@Format("yyyy") final Date from, @Format("yyyy") final Date to) {
	result = library.findBooks(from, to);
	}
	@Then("(\\d+) books should have been found$")
	public void verifyAmountOfBooksFound(final int booksFound) {
	assertThat(result.size(), equalTo(booksFound));
	}
	@Then("Book (\\d+) should have the title '(.+)'$")
	public void verifyBookAtPosition(final int position, final String title) {
	assertThat(result.get(position - 1).getTitle(), equalTo(title));
	}
	}

view raw BookSearchSteps.java hosted with ❤ by GitHub

This class name ends with Steps to denote that it provides step implementations and starts with BookSearh to denote that these steps are associated with the search_book.feature, but the association is not done with the class name, but solely relies on the regular expressions with which methods are annotated. Cucumber will search any class in the 'src/test' directory to find a method whose annotation matches a Gherkin step.

Cucumber also needs a test class that links to the Cucumber runner like:

	@RunWith(Cucumber.class)
	public class BookSearchTest {
	}

view raw BookSearchTest.java hosted with ❤ by GitHub

With Spock

Spock is a DSL for BDD written in Groovy. The search by publication year scenario can be written in Spock as:

	def 'Search books by publication year' () {
	given: "a book with the title 'One good book', written by 'Anonymous', published in 14 March 2013"
	and: "another book with the title 'Some other book', written by 'Tim Tomson', published in 23 August 2014"
	and: "another book with the title 'How to cook a dino', written by 'Fred Flintstone', published in 01 January 2012"
	when: "the customer searches for books published between 2013 and 2014"
	then: "2 books should have been found"
	and: "Book 1 should have the title 'Some other book'"
	and: "Book 2 should have the title 'One good book'"
	}

view raw LibrarySpec.groovy hosted with ❤ by GitHub

This is very similar to the Gherkin syntax and equally easy to read. The difference is that Gherkin is a text document that is parsed by the Cucumber framework and associated to Java (or other language) code, while a Spock method is at the same time a natural language specification and valid Groovy code. Therefore the implementation is done in the same Spock file as follows:

	class LibrarySpec extends Specification {
	Library library = new Library();
	SimpleDateFormat formatter1 = new SimpleDateFormat("dd MMMMM yyyy");
	SimpleDateFormat formatter2 = new SimpleDateFormat("yyyy");
	def 'Search books by publication year' () {
	given: "a book with the title 'One good book', written by 'Anonymous', published in 14 March 2013"
	addBook('One good book', 'Anonymous', '14 March 2013');
	and: "another book with the title 'Some other book', written by 'Tim Tomson', published in 23 August 2014"
	addBook('Some other book', 'Tim Tomson', '23 August 2014');
	and: "another book with the title 'How to cook a dino', written by 'Fred Flintstone', published in 01 January 2012"
	addBook('How to cook a dino', 'Fred Flintstone', '01 January 2012');
	when: "the customer searches for books published between 2013 and 2014"
	List<Book> result = library.findBooks(formatter2.parse("2013"), formatter2.parse("2014"));
	then: "2 books should have been found"
	result.size() == 2
	and: "Book 1 should have the title 'Some other book'"
	result.get(0).getTitle() == 'Some other book'
	and: "Book 2 should have the title 'One good book'"
	result.get(1).getTitle() == 'One good book'
	}
	def addBook(String title, String author, String published) {
	Book book = new Book(title, author, formatter1.parse(published));
	library.addBook(book);
	}
	}

view raw LibrarySpec_post.groovy hosted with ❤ by GitHub

The code that implemented Cucumber step methods is mixed with the natural language specification in Spock. Spock allows the implementation to be done in Groovy or Java.

The Salary Manager Test Case

With Cucumber
A more complex test case is that of a salary manager of many employees that modifies employee salaries. Although the original example considered the modification of a single employee's salary, I change the test to allow for different test input data in the same manner as a DataProvider works in TestNG:

	Feature: Salary Management
	Scenario Outline: Modify all employees salary
	Given the salary management system is initialized with the following data
	| id | user | salary |
	| 1 | donald | 60000.0 |
	| 2 | dewie | 62000.0 |
	| 3 | goofy | 55000.0 |
	| 4 | scrooge | 70000.0 |
	| 5 | daisy | 56000.0 |
	| 6 | minnie | 62000.0 |
	| 7 | mickey | 51000.0 |
	| 8 | fethry | 66500.0 |
	When the boss increases the salary for the employee with id '<id>' by <percent>%
	Then the payroll for the employee with id '<id>' should display a salary of <expected>
	Examples:
	| id | percent | expected |
	| 1 | 2 | 61200 |
	| 2 | 7 | 66340 |
	| 3 | 5 | 57750 |
	| 4 | 12 | 78400 |
	| 5 | -2 | 54880 |
	| 6 | 4 | 64480 |
	| 7 | -7 | 47430 |
	| 8 | 0 | 66500 |

view raw salary_management.feature hosted with ❤ by GitHub

To run the same test with different input data, Cucumber offers the Scenario Outline and Examples keywords. Scenario Outline denotes that this scenario (test) needs to run multiple times and the table after the Examples keyword offers the input data. The first table row contains column headers which act as variables that can be used in the steps text. The scenario runs once per data line. Notice that the first table contains initialization data, e.g. the employees that this manager manages. The same set of employees will be created for each run.

The step method implementation follows:

	public class SalarySteps {
	SalaryManager manager;
	@Given("^the salary management system is initialized with the following data$")
	public void the_salary_management_system_is_initialized_with_the_following_data(final List<Employee> employees) throws Throwable {
	manager = new SalaryManager(employees);
	}
	@When("^the boss increases the salary for the employee with id '(\\d+)' by (\\d+)%$")
	public void the_boss_increases_the_salary_for_the_employee_with_id_by(final int id, final int increaseInPercent) throws Throwable {
	manager.increaseSalary(id, increaseInPercent);
	}
	@Then("^the payroll for the employee with id '(\\d+)' should display a salary of (\\d+)$")
	public void the_payroll_for_the_employee_with_id_should_display_a_salary_of(final int id, final float salary) throws Throwable {
	Employee nominee = manager.getPayroll(id);
	assertThat(nominee.getSalary(), equalTo(salary));
	}
	}

view raw SalarySteps.java hosted with ❤ by GitHub

Notice, that the initialization data table is mapped to a List<Employee> object and passed by Cucumber as an argument of the the_salary_management_system_is_initialized_with_the_following_data method.

With Spock
The same feature specification could be written with Spock as follows:

	class SalarySpec extends Specification {
	def "Modify the salary employee #id by #percent percent" () {
	given: "the salary management system is initialized with the following data"
	[
	[1, 'donald', 60000.0],
	[2, 'dewie', 62000.0],
	[3, 'goofy', 55000.0],
	[4, 'scrooge', 70000.0],
	[5, 'daisy', 56000.0],
	[6, 'minnie', 62000.0],
	[7, 'mickey', 51000.0],
	[8, 'fethry', 66500.0]
	]
	when: "the boss increases the salary for the employee with 'id' by 'percent' % as per data table"
	then: "the payroll for the employee with 'id' should display a salary of 'expected' as per data table"
	where:
	id | percent || expected
	1 | 2 || 61200.0
	2 | 7 || 66340.0
	3 | 5 || 57750.0
	4 | 12 || 78400.0
	5 | -2 || 54880.0
	6 | 4 || 64480.0
	7 | -7 || 47430.0
	8 | 0 || 66500.0
	}
	}

view raw SalarySpec_pre.groovy hosted with ❤ by GitHub

Spock offers data tables for test input data (the analog for Cucumber Examples table), but not for initialization data. This is a clear drawback. A workaround is to define the init data as a two dimensional array. The implementation then needs to create a list of employees out of the array, as shown:

	class SalarySpec extends Specification {
	@Unroll
	def "Modify the salary employee #id by #percent percent " () {
	given: "the salary management system is initialized with the following data"
	def data = [
	[1, 'donald', 60000.0],
	[2, 'dewie', 62000.0],
	[3, 'goofy', 55000.0],
	[4, 'scrooge', 70000.0],
	[5, 'daisy', 56000.0],
	[6, 'minnie', 62000.0],
	[7, 'mickey', 51000.0],
	[8, 'fethry', 66500.0]
	]
	SalaryManager manager = new SalaryManager(employees(data));
	when: "the boss increases the salary for the employee with 'id' by 'percent' % as per data table"
	manager.increaseSalary(id, percent);
	then: "the payroll for the employee with 'id' should display a salary of 'expected' as per data table"
	Employee nominee = manager.getPayroll(id);
	nominee.getSalary() == expected;
	where:
	id | percent || expected
	1 | 2 || 61200.0
	2 | 7 || 66340.0
	3 | 5 || 57750.0
	4 | 12 || 78400.0
	5 | -2 || 54880.0
	6 | 4 || 64480.0
	7 | -7 || 47430.0
	8 | 0 || 66500.0
	}
	def employees(data) {
	data.collect { new Employee(id: it[0], user: it[1], salary: it[2])}
	}
	}

view raw SalarySpec_post.groovy hosted with ❤ by GitHub

Conclusion

Both Cucumber and Spock tight couple the natural language specification with the test code. This is a direct consequence of the BDD paradigm that both frameworks have been created to support. But Cucumber does this in a more strict manner. Changing the natural language will break the test code, e.g. with a missing step implementation if no method's regular expression matches the given step text. While for spock the text is an arbitrary string after the ':' symbol. It is not validated against the implementation that follows.

Also Cucumber offers a clear distinction between natural language specification and test code. This is an advantage for people other than developers having to write or read the specs. In the end, the very essence of BDD is the close collaboration of product owners, BAs, QAs, architects and developers so that the specification is agreed and understood by all before the development starts.

On the other hand, Spock offers a fast, concise and one file solution. Groovy's ability to use any string as a method name allows for concise test case names. It offers to developers a single point to read and understand the spec and the code that implements it, let alone the goodies with which Spock comes, like internal mock support or advanced data table capabilities (e.g. using closures in table cells).

To conclude, Cucumber looks to be a better fit for integration or end-to-end tests, that involve interaction of people of different skills and backgrounds, while Spock for unit tests, that are more or less solely developers responsibility.