Integrate Cucumber with Geb: Productive BDD Automation with Style

In this post, I show how Cucumber can be integrated with Geb for implementing end-to-end tests under the BDD paradigm, in a productive and stylish manner. As a system under testing, I choose the BBC website and test basic navigation on BBC news pages. The code is available on github.

Background

Geb is a Groovy framework for automation. It offers a powerful DSL on top of  WebDriver (aka Selenium 2). The benefit from using Geb instead of WebDriver directly is that Geb takes responsibility of the technicalities, removes the boilerplate and allows the developer to focus on the test framework he builds. It does that by exploiting Groovy meta-programming capabilities and Groovy's inherent conciseness. It has also built in support for test automation code design patterns, like the Page Object and the Loadable Component patterns, that have been proven critical for minimizing the code maintenance cost of end-to-end internal business test frameworks.

Cucumber is the most famous test framework to support the BDD paradigm. What BDD is and how much benefit agile teams can take by following it, has been discussed in one of my previous posts, about integrating cucumber with WebDriver. In this post, I follow the same objective, i.e. to automate a scenario where the user lands on the BBC home page, navigates to a news category (e.g. Business) and finally shares a video on facebook. In this post I effectively replace WebDriver with Geb and show the value of this replacement. The reader is highly recommended to read that previous post before continuing with this one, as well as comparing the two github repositories (bbc-webdriver-cucumber and bbc-geb-cucumber).

Setup

As a build and dependency management tool, I use Gradle. The build.gradle file follows. 

The Geb dependencies are geb-core and geb-junit4, as in this demo I use Junit4 (alternatives include TestNG and Spock). The Cucumber dependencies are cucumber-groovy and cucumber-junit. Also, in this demo I use the selenium-firefox-driver. Finally, the standard groovy-all dependency is added to be able to compile and run Groovy and junit to be able to use the Junit4 runner via the Cucumber runner class. Notice that some dependencies appear to be compile time dependencies, while some others testCompile. This is due to my habit when I develop test frameworks to store Cucumber Step Definition and Page Object classes under the src/ while the test runner class under the test/ directory. If someone selects to put them all under the test/ directory, then the testCompile status is enough for all the dependencies.

BBC News Feature

The demo scenario is similar to the one presented in the bbc-webdriver-cucumber demo, but is simplified and written in user terms.

Geb Page Objects

Due to its importance in writing effective and maintainable test frameworks, the Page Object pattern has built in support on Geb. In a sense, Geb forces the developer to follow this pattern, while WebDriver leaves it to his decision/knowledge. Let's look at NewsPage class to see how a Geb Page class looks like.

As one can observe the NewsPage class extends the geb.Page class and defines three static variables, url, at and content. The first one is of type String and defines the url of the web page modeled by the NewsPage class. The second one, i.e. at, is of type Closure and defines a boolean post-condition of the web page loading, i.e. a condition whose true value makes us sure the web page has been loaded correctly. This is normally a check to see if some representative web element is displayed. The content variable is of type Closure, which in turn contains a set of closures, each one representing a web element. For instance, the news closure represents the "NEWS" text in the upper left corner of the BBC news page that is written with white font on top of a red background. The news closure contains a locator for that web element. Geb uses a very powerful JQuery-style locator syntax. Although Geb allows you to work with the org.openqa.selenium.By static methods for defining locators, the gebbish JQuery-style syntax is much more flexible. When called, this closure returns a proxy of the locator. Similarly to WebDriver PageFactory functionality, Geb's locators are lazy, in the sense that they are resolved against the active web page the first time a method is called on them. For instance, when someone calls the at closure, the following steps take place:

1. news closure is called 
2. news closure returns a proxy of the web element located by the specified locator.
3. the method isDisplayed() is called on that proxy
4. the web element located by the specified locator is initialized (NoSuchElementException can be thrown if no element is found) 
5. isDisplayed() returns true or false, depending on if the web element is currently displayed or not on the active web page

It might not be obvious to someone unfamiliar with Groovy, but news.displayed is equivalent to news().isDisplayed(). If a closure has no argument you can omit the parentheses. Also, the Java-style named getters getX() for a field x (or isX() if x is of boolean type) are equivalent with x in Groovy.

The second, i.e. categoryLink, closure defined inside the content closure is even more interesting. Since categoryLink is a closure, it can take parameters. This gives you the very powerful ability to define a group of locators, that differ only in the specified parameters. For instance, categoryLink can locate all the links in the BBC news page navigation bar. Each one of these links points to a different news category page, e.g. Business, Politics etc. All those links differ only on their text (and of course on the url they point to) and therefore can be defined in a unified manner.

Finally, NewsPage class defines a method for navigating to the NewsCategory page representing the specified category. This method returns the NewsCategorPage object following the "strong" page object pattern, that requires for any method to return the page object that will be active after its call. The call categoryLink(category) returns the proxy of the link located by the locator corresponding to the specified category and then the click() method is called on this element to navigate to the specified category web page. At this point we need to notify Geb that the active page object has been changed. This is done with the browser.page(new NewsCategoryPage(category: category)) call. A lot of things take place here, so let's examine this call in detail.

First a new NewsCategoryPage object is created and its category field is initialized with the specified category string. Then, this object is passed as argument to browser.page method. The browser object is a field of the superclass geb.Page. Its type is the Browser class, which is effectively a wrapper around a WebDriver object. By default Geb uses the FirefoxDriver, but this is configurable. The browser object exposes the driver object, e.g. you can access it by writing browser.driver (which is equivalent with browser.getDriver()) and work with this directly but in most cases you don't need to do this. You may notice that there is no return statement in the selectCategory method. This is a famous Groovy idiom, the method returns the object that the last expression is evaluated to. In this case, browser.page(new NewsCategoryPage(category: category)) returns the page object that is passed as argument, which is of NewsCategoryPage type.

Steps Implementation

The Step Definition file follows.

As one can easily observe, there is no class, step definitions are written in a script form. In Groovy, classes are optional, meaning that Groovy can be used as a scripting language. Before and After are cucumber Hooks that run before and after each scenario since I don't pass any parameter (they could run only before/after some scenarios specified by their parameters). 

The Before hook block deserves some explanation. With the BindingUpdater class, we inject a Browser object in this script. BindingUpdater is a Geb class which wraps around the Groovy Binding class. This is normally used in Groovy scripts to inject objects. Any programming language supports some concept of Binding (some of them call it "environment"), e.g. if you define a variable x = 5, then the language at runtime creates a variable and bounds it to the value "5". After running the Before hook, each scenario has access to the injected browser object, which has been bounded to an instance of Browser class. 

For instance, the first step implementation block is to HomePage. In Groovy, parentheses are optional, if a method has a single parameter, e.g. to HomePage is equivalent with to(HomePage). Second, in Groovy classes are first class citizens, e.g. to refer to HomePage class, you can write HomePage, instead of HomePage.class (the ".class" is optional). Since there is no method named to taking as argument the HomePage class, one could ask how this script runs successfuly. The trick is that such a to method is defined in the browser object that has been injected to the script. Therefore, to HomePage is equivalent with browser.to(HomePage.class). You could write it like that, but the Gebbish point is to get rid of the boilerplate and stick to the concise and stylish to HomePage syntax. The idea is you should focus on your test goal which at this point can be described as "go to BBC home web page". All the rest are technicalities that Geb handles for you.

If you read this script, you will realize I only used the to method to load the first web page, which is the BBC home page. For all the rest navigations, I followed the "strong" page object pattern with calling methods exposed by page objects to navigate to other web pages, like page.selectLatestNews() which navigates from the BBC home page to the BBC News page. Each time the page object (this is equivalent to browser.page) is the page object corresponding to the active web page. This is the result of me following the "strong" page object pattern. The page object is defined with type geb.Page in the Browser class, which does not define any selectLatestNews method, but Groovy supports a dynamic type system (aka duck typing) by default (you can switch to static typing by using annotations though), i.e. it resolves the object types at run time. As a developer, I know that at this point the page object is of type HomePage and therefore has a method selectLatestNews. In the following line, I check if the active web page is the news category page to assure that this web page has been loaded correctly. This will invoke the at closure defined in the NewsPage class. 

Notice the difference with the following step implementation that includes def newsCategoryPage = page.selectCategory(category). This call implements the navigation from the news page to the news category page for the category specified in the feature file. Since this step can navigate to different category pages an at call with parameter the NewsCategoryPage class is not enough. NewsCategoryPage class models all the news category pages, while I want to be sure that the active web page is the one corresponding to the category specified in the feature file. To this end, I need to assign the page object returned by selectCategory and pass this as the at method parameter. Browser class overloads the at method, so that you can either pass a Page class (or list of potential Page classes) or a Page class instance (or a list of potential Page class instances).

Furthermore, notice that the first step implementation don't need to make an at call, since the to method does an at call at the end to check if the target web page has been loaded correctly.

Finally, all step definition "methods" are actually not methods but calls to Cucumber methods that take as argument a string pattern and a closure. If the final argument of a method is a closure, then Groovy allows this to be written outside of the parentheses. For instance the first step, i.e. 

Given(~/^the user is landed at the home page$/) { -> to HomePage }

is equivalent to 

Given(~/^the user is landed at the home page$/, { -> to HomePage }).

Runner Class

The Groovy version of the Cucumber Runner class is very similar to the Java version:

Run this on eclipse with "Run As"->"JUnit Test" or by command line with gradle clean test.

Outro

To sum up, Geb boosts the productivity of the automation developer beyond any preceding level. To achieve this it allows you to focus on what really matters, the web elements of pages under test and the interaction between web pages as defined by your product requirements. Gebs manages all the technical details for you. Also Geb forces developers to follow design patterns like the Page Object and the Loadable Component patterns which are critical for the minimization of the code maintenance cost. Since it is heavily relied on the Groovy meta-programming features, it requires the developer has some level of maturity with Groovy. The JQuery locator syntax might look unfamiliar to back-end developers but is really easy to learn and Geb has excellent documentation (not only about the locators). It integrates seamlessly with Cucumber and together they make a perfect combination for developing end-to-end test frameworks that require web page automation and are developed by agile teams following the BDD paradigm.

Note 04/02/2017:

I have upgraded to Selenium 3. 

You will have to download the geckodriver, unzip it and set the path to it on src/test/resources/GebConfig.groovy.

The project was tested with: 

• selenium version 3.0.1
• geckodriver 0.14.0
• firefox 51.0.1

XML Transformation: XSLT vs Groovy

In this post I show how Groovy can be used for XML transformation, instead of XSLT. The idea is to combine XmlSulper with MarkupBuilder for a powerful and concise syntax for XML transformation.
Everything that is presented in this post is uploaded on github.

XML Transformation Example

As a demo XML input, I use the Sky Sports Football News Feed. This contains a list of news items, as shown below.

Let's say we would like to transform this XML into an HTML file with a table of two columns, i.e. the timestamp (news:publication_date element) and the link (loc element) accompanied with the title (news:title element), like the following HTML snippet.

The XSLT Case

You could do this transformation with the following XSL file

To check the effect of this XSL stylesheet on your own, I have downloaded the XML input into sitemap_news_football.xml and have added the following line
<?xml-stylesheet type="text/xsl" href="skyFeed.xsl" ?>
in order to point to the skyFeed.xsl stylesheet. Clone the gihub repository and just open the sitemap_news_football.xml on the browser of your wish.

The Groovy Case

You can do the same transformation with the following Groovy script.

To run this script you need to have Groovy installed and the groovy binary in your path. Run:
> cd xslt-vs-groovy 
> skyfeed sitemap_news_football.xml

This will create the output file out.html in the same directory. Open it on your browser to verify the result is the same as opening sitemap_news_football.xml on the browser. You can also run the script with no input file argument:
> skyfeed

This will consider as input the current XML feed content.

The magic happens by blurring XmlSlurper with MarkupBuilder.

XmlSlurper's parse method returns a GPathResult object (e.g. the object referenced by root in this example) and then you can navigate over XML elements inside your Groovy code with GPath. GPath is a path expression language integrated into groovy (as a Groovy DSL). Its syntax is very similar to XPath, but has an object oriented flavor, e.g. instead of '/' it uses '.'. For instance, root matches the XML root element, which is the urlset elem. Similarly, root.url matches all the url elements that are children of urlset. Groovy evaluates root.url into a NodeChildren object, which then can be iterated over with a groovy closure as shown in the script.

MarkupBuilder allows you to produce HTML (or XML) by using normal Groovy contracts, like methods -more precisely missing (not implemented) methods which are delegated to methodMissing method- and closures. For more details on Builders, you can reed "Chapter 7: Building a Builder" of "Groovy for Domain-Specific Languages".

One should consider though, that browsers do not run Groovy, while they do run XSLT processors. This means that XML transformation with Groovy can only happen at the server side and then you need to send the output over HTTP to the client. If you need to do the XML transformation in the client side you need to stick with XSLT.

Groovy Shell Scripting Meets Hadoop

In this post I show how Hadoop map and reduce functions can be written as Groovy shell scripts in a concise and elegant way.

The most common way to kick of a hadoop map-reduce task by command line is a python script. Alternatively, one could write the map and reduce functions in java, but he has to pay the additional cost of packaging them in a jar and running the jar in the command line. The difference between python and java is that python is a scripting language, while java not. For instance, you can run directly python as a script, by adding "#!/usr/bin/env python" at the top of your files.

Groovy fills this gap, being the first scripting language among the family of jvm languages. Reasons to prefer groovy from python inlcude:

1. You are a java/groovy developer and you want to write the map reduce functions in a familiar language;
2. You want to leverage the power of jvm as well as of a vast amount of libraries written in any jvm language;
3. You find python's syntax pathetic, e.g. change indentation and you change the code semantics;
4. Groovy goodies allow you to quickly write concise and elegant code.

Requirements

For running hadoop, I use the cloudera vm (you can download it here for free). Of course, you can use the hadoop infrastructure of your wish. Hereafter, I refer to cloudera vm and you need to adjust properly if you use an alternative.

Groovy needs to be installed in the machines that run hadoop. It is not on cloudera vm, so you need to do it, read instructions here.

The Wordcount Example

This is probably the most popular "hello word" hadoop example. The problem is stated as follows: Given a collection of files that contain strings separated by blank spaces, compute the number each word occurs across the collection. The output should be a file with records of the form <word #occurences>.

The mapper groovy script follows.

The logic is clearly communicated by the code: for each line read from the standard input, trim it, get all the words (assuming they are separated with each other by blank spaces) and for each word emit  "word    1" (a tab separates the key from the value), meaning that we found one occurrence of the word. The first line #!/usr/bin/env groovy tells the OS to use groovy to interpret this file.

The reducer groovy script follows.

Again, the logic is straightforward: for each line read from the standard input, trim it and get the word, #occurence fields out of it. If the current word is different than the previous emit the record "word counter" and reset the counter. In any case, update the previous word with the current one and sum the current value (#occurence) to the total counter. In the end, emit the record for the last word.

You need to make these 2 scripts executable:
chmod +x wordcount_mapper.groovy
chmod +x wordcount_reducer.groovy

Assuming, you have put the input files under the hdfs directory '/user/cloudera/wordcount_input', you can run the hadoop with:

hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming.jar -input /user/cloudera/wordcount_input -output /user/cloudera/wordcount_output -mapper wordcount_mapper.groovy -reducer wordcount_reducer.groovy

If you get a failure including the message "Container killed on request. Exit code is 143", this means that groovy is not visible by the datanode (cloudera vm includes just one data node) that needs to run the map or reduce jobs. You may have installed groovy on the cluster node, but you need to install it on all data nodes too, as hadoop pushes computation to them.

A workaround for cloudera vm is to replace "groovy" in the first line of your scripts with the absolute path of the groovy file. To find where groovy is installed, run: 
printenv GROOVY_HOME

For instance, if you have installed it with sdkman, groovy is located under '/home/cloudera/.sdkman/candidates/groovy/current/bin/' and you have to replace the first line of both scripts with:
#!/usr/bin/env /home/cloudera/.sdkman/candidates/groovy/current/bin/groovy

Although not needed for this problem, you can reuse any published jvm library. Grape will fetch the dependency from a repository and you need to add import statements as you normally do in java. An example can be found here.

LP-reduction: A Groovy Framework for Linear Programming Reductions

In this post I present a Groovy framework for Linear Programming (hereafter LP) reductions. The term 'framework' denotes that its expressive power covers the reduction of any problem P into LP.

Given a Problem P with a domain D and range (or solution domain if you wish) R, we say that a problem P reduces to LP, if you can use an algorithm that solves LP to solve P. The reduction includes 3 steps (a nice visual representation can be found at page 9/46 of this):
1. given an instance x of P, map it to an instance y of LP;
2. solve y by an algorithm (in this case Simplex) for LP, to derive a solution, let's denote it with LP(y);
3. map LP(y) to the solution of x.

The mapping functions that are needed by steps (1) and (3) are passed as inputs in the form of Groovy closures, while step (2) is delegated to Apache Math3 Simplex implementation by default, although abstractions for replacing this with the solver of your wish are provided.

Of course, not any possible combination of map functions in step (1) and (3) lead to a reduction. You need to prove that a certain combination is proper. The framework presented here does not verify the combination is proper (there are certain limitations even if we wished to do so), but assumes it is proper.

Background

Generally, algorithmic reductions are very useful for both theoretical and practical reasons. For instance, if A reduces to B polynomially (i.e. the mapping functions are polynomial) and B is a polynomial problem, then A is polynomial as well (for more details about the significance of reductions you can read this or this). In addition, if you have an efficient implementation for an algorithm B, but not for A, you can leverage this implementation.

Specifically for B=LP:
1. LP is polynomial. The ellipsoid algorithm solves this polynomially, although the simplex algorithm (which is exponential in the worst case) is much faster in practice.
2. The expressive power of LP and its massive application in the financial sector (e.g. see operations research) has led to very efficient, commercial simplex implementations offered by many business analyst vendors, like IBM.

Source code, Dependencies, License

The lp-reduction framework is an open source project licensed under Apache License, Version 2.0. 

The code is hosted by github under the lp-reduction repository.

Gradle is used for building and dependency management.

The only compile time dependency is the Apache Commons Mathematics Library, while testing is done with Spock.

How it works

The implementation is straightforward as the main class (LPReduction) suggests:

Step (1) is implemented by 'mapToLP' method. It takes as input an instance x of P and returns the LP instance y. To do so it calls the 'vars'/'objFunc'/'constraints' closures that are defined as class fields in order to create the LP variables/objective function/constraints respectively. How they are initialized will be clarified later with an example.

Notice that the LPInstance class has two more boolean fields, namely 'max' and 'nonNegative'. The former indicates whether this is a maximization (default option) or a minimization (set max=false for this) problem.

Moreover, it is possible that an LP instance has empty (technically any constant, usually 0) objective function and in this case it is often called constraint satisfaction problem. This happens when you only need to know if there is a solution that simultaneously satisfies all the constraints, without having any quantity to maximize/minimize. This case is handled by the line:
objFunc ? objFunc(x) : new LPFunction([:], 0)
which means "if the closure objFunc is set (i.e. not null), call it, else use the empty objective function (i.e. f(x)=0)".

Step (2) is implemented by 'solve' method, which takes as input an LP instance y and returns its solution. If no solver is set (default option), the Apache Math3 Simplex is used to solve y, else the user provided solver is used. To allow this LPSolver interface provides the proper abstraction:

The solution is modeled by Solution class:


The whole reduction is implemented by 'reduce' method. Obviously, it runs the first two steps as explained above and then it calls the 'result' closure to map the solution of the LP instance y to the solution of P for x.

The Max Flow Reduction Example

The code below exercises the lp-reduction framework for reducing the Max Flow problem to LP. You can read here what the Max Flow problem is and how an instance of it can be mapped to an LP instance.

As shown above, you can initialize an LPReduction object fields with the 'with' Groovy keyword. After defining the LPReduction object, you can exercise it as shown in MaxFlowSpec.groovy. Notice that the code inside the closures vars/objFunc/constraints/result can be written in Java instead of Groovy. However, you can not replace the entire closure with a Java 8 lambda as Groovy is not compatible with lamdas.

The full code can be found github under the lp-reductions repository.

The Graph Realization Reduction Example

The following code exercises the lp-reduction framework for reducing the Graph Realization problem to LP.

Notice that in its original version the Graph Realization is a decision problem, i.e. is there any graph with the given sequence, YES or NO? In practice, it is often useful to ask for a graph with the given sequence if such a graph exists. To define the decision version you need to define the result closure as:
result = { Solution s-> s.state==LPIState.FEASIBLE }
while for the modified version you need to define it as
result = { Solution s -> s.state==LPIState.FEASIBLE ? s.at : null }

After defining the LPReduction object, you can exercise it as shown in GraphRealizationSpec.groovy.

Groptim: A Groovy DSL for Linear Programming

In this post I present how Groovy meta-programming features can be leveraged to implement a Linear Programming (hereafter called LP) DSL, called Groptim. Internally, Groptim uses the Apache Commons Mathematics Library and its Simplex implementation (see org.apache.commons.math3.optim.linear.SimplexSolver) to solve the LP instances.

Source code, Dependencies, License

Groptim is an open source project licensed under Apache License, Version 2.0. 

The code is hosted by github under the groptim repository.

Gradle is used for building and dependency management.

The only compile time dependency is the Apache Commons Mathematics Library, while testing is done with Spock.

Usage for Groovy Developers

Plenty of examples on how to use Groptim from Groovy code can be found at LPSolverSpec.groovy. In this paragraph, I pick some representative examples to present the flexibility and the limitations of the syntax.

Every LP instance can be written in the LP canonical form, which defines all the constrains to be "less than or equals to" and all the variables non-negative:

Some flexibility beyond the canonical form is offered:

1. you can define min instead of max to minimize the objective function;
2. constants are allowed in the objective function; 
3. variables are allowed to be negative (if you want one to be strictly non-negative, you need to make this explicit with a constraint).

Groptim strictly adheres to the canonical form for constraints. All the constraints are deemed "less than or equals to", no matter what the comparison symbol is ('<=', '>=' or '='). The reason for this decision will be described in later sections.

Let's see how an LP instance looks like in Groptim:

The first line obviously creates an object of the main Groptim class, which is LPSolver.groovy. To define a maximization problem we write lp.max (for minimization write lp.min).

Immediately after the max (or min), we have to define the Objective Function (hereafter OF). in brackets {} (the space between max and the brackets is optional). In this case the OF is 15*x0 + 10*x1 + 7. Notice that we write 7.c to denote the constant number 7. This is needed as every term of the OF (and itself as a whole) has to be evaluated as an Expression object by Groovy. We will dive into implementation details in the next paragraph. Notice that constants are not needed in the OF. Finding a vector x that maximizes (or minimizes) 15*x0 + 10*x1 + 7 is equivalent with finding a vector x that maximizes (resp. minimizes) 15*x0 + 10*x1. This is why the canonical LP form does not allow for constants in the OF. For Groptim a constant in the OF is optional.

After the objective function block, the keyword 'subjectTo' has to follow to denote that the block that follows defines the constraints. Instead of 'subjectTo', one can write 'subject() to', but not  'subject to'. Looking at the constraints, the first two are straightforward, but the 3rd and 4rth are the result of canonizing the non-canonical constraint x0 + x1 = 4. This is equivalent with x0 + x1 <= 4 and x0 + x1 >= 4 that should both be satisfied. The first is canonical, the second can be easily canonized by multiplying -1 both sides of the inequality and therefore changing '>' with '<': -x0 - x1 <= -4.

That's it, this is valid Groovy code. After the subjectTo block you can access the lp fields to read the solution of the LP instance, e.g.:

which means that the OF is maximized at (x0, x1) = (2, 2) and its maximum value is 57.

Notice that any string (except for Groovy reserved keywords) can be used for variable names, e.g. instead of x0, x1, x2, one can write x, y, or myvar. Also coefficients can be arbitrary real numbers and standard arithmetic rules apply, e.g.

• 0*x1 is equivalent with 0.c
• 1.7*x0 - x1 is equivalent with 1.7*x0 + -x1

Also, non-linear OF, or constraints are not allowed in LP, therefore Groptim throws an IllegalStateException if you write a something like x0*x1 or (2*x0 + 5.c)*(4*x1-3.c).

Usage for Everyone

One of the many reasons why Groovy is "groovy", is that it can be used as a scripting language. In the script/ directory I have included the groptim script along with two input LP files written in Groptim:

Notice that the lp object is not needed in this case. I will explain in the following paragraph how the script silently injects it. You can run groptim script to solve the LP instances described in these files and get the solution printed in your console as:

To allow this magic to happen, you need to install Groovy and have the groovy command in your path. Notice that groptim script has 2 dependencies (managed by Grape), apache math ('org.apache.commons:commons-math3:3.5') and Groptim itself ('org.ytheohar:groptim:1.0'). Both will be fetched from the maven central repository. When groptim script runs it will have Ivy copy these artifacts under your .groovy/grapes/ directory.

How it works

The objective function as well as both the left and right part of each constraint is evaluated as an Expression or more precisely as one of its subclasses, i.e. NumberExpression, LPVar or BinaryExpression. Take for instance 15*x0 + 10*x1 + 7.c:

1. x0 is evaluated as an LPVar
2. 15*x0 as a BinaryExpression
3. x1 as an LPVar
4. 10*x1 as a BinaryExpression
5. 7.c as a NumberExpression
6.  15*x0 + 10*x1 as a BinaryExpression
7.  15*x0 + 10*x1 + 7.c as a BinaryExpression

The simplest form of an expression is a NumberExpression. To evaluate (step 5) a constant into an expression I had to add a new method getC in the java.lang.Number class. This is done in the LPSolver constructor as follows: 
Number.metaClass.getC { new NumberExpression(delegate) }

Then, we can append any number with .c and Groovy will evaluate this by delegating to the method getC, e.g. 7.c is equivalent with 7.getC() which in turn is evaluated into new NumberExpression(7).

The evaluation of x0 and x1 (steps 1, 3) as LPVar objects is more interesting. This is a good point to introduce the LPSolver class which implements the core of the Groptim DSL:

The max method takes as argument the OF block, which is evaluated in a closure by Groovy and passes that closure to optimize method. This changes the delegate object from the object that owns the closure (i.e. the one it was defined in) to this (i.e. LPSolver instance). Then it calls the closure, which triggers the above evaluation of the OF block into a BinaryExpression.

On the first occurrence of x0 in the OF block, Groovy will search for a field x0 in the delegate object (which is an LPSolver instance). It will not find any, so it will call the propertyMissing method on the delegate object. The implementation of this method first looks if a variable with the name 'x0' has been registered before and if not it registers it. It is important to understand that if I had not set the delegate to be this in the optimize method, then Groovy would have called the propertyMissing method on the object that owns the LPSolver instance, possible throwing a MethodMissingException. This would be client's class, e.g. it wiould be LPSolverSpec, when we run a test.

Both LPVar and NumberExpression define a negative() method, to which Groovy delegates the symbol '-', e.g. -x0 (strictly no space between '-' and x0) is equivalent with x0.negative() and -7.c to 7.getC().negative().

A number multiplied with an LPVar is evaluated (steps 2 and 4) as a BinaryExpression. To do this, I had to add the multiply method into the Number.metaclass (see LPSolver constructor). This makes 15*x0 equivalent with 15.multiply(x0) which evaluates in a BinaryExpression.

Groovy delegates the symbol '+','-' (when there is one or more spaces between '-' and a variable, e.g. x0) and '*'  to plus, minus and multiply method respectively (steps 6, 7). It should be no surprise to you that Expression class defines all of them, to make possible the composition of Expressions (BinaryExpressions to be more precise) from other Expressions:

Finally, Groovy delegates all the comparison operators ('<','>','<=','>=') to the compareTo method, e.g. x0 + x1 <= 4.c is equivalent with (x0 + x1).compareTo(4.c) <= 0. Notice that the compareTo implementation creates a new constraint and registers it in the solver object. This poses a limitation for Groptim as any comparison operator will be delegated to the same method and there is no way to know which comparison operator triggered this call. Therefore, any constraint is considered as '<=' constraint from Groptim and this is the reason why Groptim strictly follows the canonical LP form for constraints.

It should be stressed that Groovy delegates the '==' operator into the equals method (e.g. x0 == 2.c is equivalent with x0.equals(2.c)), except if the delegate's class implements the Comparable interface, in which case it delegates to compareTo (e.g. x0 == 2.c is equivalent with x0.compareTo(2.c) == 0). Therefore, Groptim will consider a '==' constraint as an '<=' constraint too.

More precisely, if the delegate class implements the Comparable interface, the '==' operator is not directly delegated to the compareTo method. Take for instance, the constraint x0 == 2.c. If neither x0 (LPVar instance) is of a subtype of 2.c (NumberExpression instance) or 2.c is  of a subtype of x0, then Groovy will not call compareTo at all, as it can conclude that the two objects are not equal. This check is done in the Groovy SDK class org.codehaus.groovy.runtime.typehandling.DefaultTypeTransformation and its compareToWithEqualityCheck method:

Due to this reason, I defined both LPVar and BinaryExpression as subclasses of NumberExpression. Otherwise, compareTo would not be called for '==' constraints and I would loose them, since constraint objects are created in the compareTo method.

Another interesting point is the return type of the optimize (and therefore max and min) method. It is a map that maps the String 'subject' to the closure { it -> ['to': { Closure cTo-> subjectTo(cTo)}]} and the String 'subjectTo' to the closure { it -> subjectTo(it)}. As you may have realized, this allows for the two alternative Groptim syntaxes for constraints, i.e. 'subject() to {}' and 'subjectTo {}' respectively.

When you write 'subjectTo {}', effectively you ask for the value that 'subjectTo' maps to. This value is a closure that takes as argument another closure denoted by it (represents the constraints block) and passes that to the subjectTo method. Pure fun! Isn't it?

The case that you write 'subject() to {}' is a little more interesting (more fun). In this case, you effectively ask for the value that the String 'subject' maps to, which is a closure. By writing () you call this closure which returns the map ['to': { Closure cTo-> subjectTo(cTo)}]. Obviously, this maps the String 'to' to another closure {Closure cTo-> subjectTo(cTo)}. This closure is returned when you write 'to'. In turn this takes as argument another closure (denoted by cTo) which represents the constraints block. This is called command chain. In Groovy, when you call a method with arguments, the () can be omitted (e.g. 'subjectTo {}' is equivalent with subjectTo({})), but when the method has no arguments, () is mandatory. This is why you can not write 'subject to {}'.

Reducing Boilerplate when Running as a Script

Groovy offers a GroovyShell class with which you can run Groovy code as if it was a script. One can initialize a GroovyShell object with a Binding object, which implements what is mostly known as 'Environment' in programming languages, e.g. a set of bindings in key-value pairs. I used this mechanism to inject the LPSolver object to any input file, so that the groptim script user does not need to create the LPSolver object. Take a look at the groptim script code:

Whenever an input file writes max (min), this is delegated to lp.max (lp.min) method. This is achieved by binding the max (min) keyword with a method reference to the max (min) method of the lp object. By calling the evaluate method on the GroovyShell object you can then evaluate Groovy code, that can either be contained in a file passed as argument (as in this case) or directly passed as an argument in the form of a String.

Internal Simplex Engine

By default (see the no-argument LPSolver constructor) Groptim uses the Apache Math Simplex implementation to solve LP instances. However, one can replace it with any other implementation as long as it implements the SolverEngine interface.

Not as 'final' as you think

The final keyword is used on methods to prevent subclasses from overriding them and on classes to prevent subclass definition. The idea behind it, is to prevent the class behavior modification. In many cases, this requirement comes to enforce security, e.g. with String objects passed to Class.forName(), or used as keys of a HashMap. But how 'final' these classes are? This posts shows they are not as 'final' as you think. More precisely, I show how you can:

• add methods to final classes;
• override/redefine methods of final classes;
• override/redefine final methods.

To achieve these goals two different approaches are followed, namely a) exploiting the meta-programming Groovy features and b) exploiting the Byte Buddy tool in Java for byte code manipulation.

The Groovy Way

The code presented in this section is on github. With Groovy you can add or redefine methods, e.g. I am going to add methods on the final java.lang.String class and override its startsWith method.

Let's take a closer look at the addMethod method. String.metaclass is equivalent with String.class.getMetaClass(). In Groovy classes are first class citizens and so you don't need to write the .class ending. String.class is an object of type Class and each Groovy object has an associated Metaclass object. All Groovy classes implement the GroovyObject interface, which exposes a getMetaClass() method for each object.

With String.metaClass.someMethodName = {...} you add a new method called someMethodName into the Metaclass object of String class. Its implementation is included in the closure (the '=' symbol is optional). Also groovy allows for String variables to be used as method names, e.g. if I have a String s and another String name="length", then s.length() and s."$name"() are equivalent. So I believe you are now able to understand that addMethod adds a method on the String class that

• is named after the name parameter;
• has no parameters;
• returns the String "funny method: String."+ name (the return keyword is optional in the last command of a method/closure).

From Groovy for Domain-Specific Languages:

"A method invocation on an object is always dispatched in the first place to the GroovyObject.invokeMethod() of the object. In the default case, this is relayed onto the MetaClass.invokeMethod() for the class and the MetaClass is responsible for looking up the actual method."

The same mechanism is also used in overriddeStartsWith method. But since a method startsWith: String => boolean already exists in String class this will be hidden. Instead the implementation we set in the closure will be called by the MetaClass.invokeMethod().

You can exercise the above code with Spock:

Run the tests with Gradle:  gradlew test

When people from the Groovy community present this short of power in public, they often get the question: "does Groovy open a security hole into Java?". This question is justified by the fact that Groovy code can be called by Java code as they both translate into JVM bytecode. Groovy guys, usually respond that groovy is entirely written in Java, so it does not add a security issue, but exposes what already existed in Java. Clever answer, certainly says the truth but not the whole truth, as is shown in the following paragraph.

Leverage Groovy Magic to Redefine Methods when Running Java?

The above paragraph, was about how to redefine methods of final Java classes, when you launch the jvm with the groovy command. This paragraph is about the same but when you launch the jvm with the java command. In this case the Metaclass mechanism is not there and therefore a method invocation invokes the class method directly.

I have prepared another test project to show this, clone it from github. This uses the Groovy code presented in the previous paragraph as a dependency. So you need to install it in your local maven repo. To do this run from the command line (assuming you have cloned the groovy project):

• cd not-as-final-as-you-think
• gradlew publishToMavenLocal

The following Java test calls the addMethod of the StringModifier class which was written in Groovy.

The test passes as a NoSuchMethodException is thrown when calling the getDeclaredMethod method on String.class.

Even if Groovy code can be called from Java, the Groovy meta-programming "magic" is lost. Groovy can't help in achieving our goal as long as we run java. Instead we need to use a byte code manipulation tool. I will use Byte Buddy, as this is the most easy and flexible byte code manipulation tool I am aware of.

With Byte Buddy

Let's say we have a final Java class with a final returnFalse method as follows:

Then we can use Byte Buddy, to redefine the returnFalse implementation, e.g. to make it return always true. This can be done as follows:

The next question is can we do the same for a method of String or any other of the final JDK classes? E.g. would the following work?

The above code attempts to redefine the isEmpty method of the String class so that it always returns true. Compare redefineMethodOfStringClass with redefineFinalMethodOfAclass. They are almost identical, but redefineMethodOfStringClass throws a NullPointerException in line:
.load(String.class.getClassLoader(), classReloadingStrategy);

This is due to the fact that JDK classes are loaded with the Bootstrap Classloader which is not exposed to the developer. Therefore String.class.getClassLoader() returns null. Byte Buddy reloads the class whose method is redefined with a custom classloader. To do this it needs the parent classloader so that all the class dependencies can be resolved by the custom classloader. Notice that this is not a Byte Buddy's limitation, this is inherent for any tool that reloads classes.

We conclude that in Java it is possible to change the behavior of final methods, or methods of final classes, for all classes except for those included in the JDK.