OroCommerce comes with a multitude of features that you can configure in a variety of ways to fulfill your specific business needs. To ensure all the built-in features work as expected, no matter what configuration combination business is using, Oro developers write tests. Since OroCommerce has many features, automated testing is a big part of the development. From the project start, we have been writing unit and functional tests, which help to test the application architecture and programming APIs.
However, these kinds of tests give developers no guaranty that when the user opens a page, everything is going to work as expected. This means that we need a different kind of tests, which instead of relying on the source code behaves like an actual user.
After an in-depth investigation, we decided to use Behat Behavior-Driven Development framework for this kind of tests as it can emulate the user very well, run tests in a real web browser, and most importantly, it uses a business-readable, domain-specific language called Gherkin to describe tests.
Using Behat Framework for Writing Tests
Here is an example of the “Sign in to the storefront” feature in the Gherkin language:
Feature: Sign in to the storefront
Scenario: Successful login
Given I signed in as AmandaRCole@example.org on the store frontend
Then I should see text matching "Signed in as Amanda Cole"
Scenario: Unsuccessful login with the wrong password
Given I am on the homepage
And I click "Sign In"
And I fill the form with:
| Email Address | NotExistingAddress@example.com |
| Password | test |
When I click "Sign In"
Then I should see "Your login was unsuccessful. Please check your email address and password before trying again."
The snippet above is a real ready-for-use test with two test cases. What’s more, business analysts can now read these tests and even write them by themselves. It’s the main benefit of Gherkin over other frameworks where tests are written in a programming language like PHP.
How Behat Tests Work
During execution, Behat tests open the Chrome browser and, similarly to a real user, follow links, fill in forms, click on buttons, scroll down pages, and so on. When there is a hidden menu link in a folded menu, Behat unfolds this menu to see a hidden link and clicks on it emulating actual users even further as they can’t normally click on hidden links in real life. As a result, Behat tests always look like real test scenarios that you can follow manually yourself. You can even use them as part of your project documentation.
Describe Test Steps with Feature Contexts
Behat scenarios are executed line by line. To make the test framework understand what you are writing, developers have to explain what to do in plain PHP when the test framework runs every line of the test. These instructions are grouped into FeatureContexts. Here is an example of one:
* Open dashboard login page and login as existing user
* Demo user should have password the same as username, e.g. username: charlie, password: charlie
* Example: Given I login as administrator
* Example: Given I login as "charlie" user
* @Given /^(?:|I )login as "(?P<loginAndPassword>(?:[^"]|\\")*)" user$/
* @Given /^(?:|I )login to dashboard as "(?P<loginAndPassword>(?:[^"]|\\")*)" user$/
* @Given /^(?:|I )login as administrator$/
* @Given /^(?:|I )login to dashboard as administrator$/
* @param string $loginAndPassword
public function loginAsUserWithPassword($loginAndPassword = 'admin')
//quick way to logout user (delete all cookies)
$driver = $this->getSession()->getDriver();
With regular expressions in the @Given, @When or @Then annotations, developers tell how to match the line in a Behat feature. The method accepts arguments from the parsed expression and the method body explains what actions to perform in PHP.
PhpStorm Behat Plugin
We use the PhpStorm Behat plugin in order not to memorize all available contexts while designing a scenario in a feature. When you start typing a keyword, PhpStorm shows you hints on the implemented steps that match the keywords, e.g., when you type grid or form, the steps that involve these items pop up in the suggestions block.
Run Behat Tests
Once the Behat feature is completed, save it in the bundle in the /Tests/Behat/Features/ folder. To run the feature, use the command below. There, the argument is the path to the feature inside the project:
Behat Tests in OroCommerce
There are 900 context methods used for parsing 40000 steps organized in 4200 scenarios combined into 840 features.
All Behat tests in OroCommerce and OroCRM run for 19 hours on fast, optimized Google Cloud Engine instances. To get feedback from CI in 15 minutes, tests are running in parallel on 75 GCE instances.
At Oro, Behat tests cover 100% scenarios in all new features and detected bugs. As a result, three years on since starting using Behat, we have created FeatureContexts for practically all possible cases when using OroCommerce. For instance, Behat knows how to fill in custom data grid filters and even read an email when it’s a part of the user scenario. So in most cases, our quality assurance team writes new tests without involving the development team.
Behat Tests in Customizations
When you’re using a vanilla OroCommerce application, you don’t need to run tests by yourself since Oro’s powerful CI servers are already testing all the possible scenarios.
However, when you are customizing the built-in feature or writing a new one, it always makes sense to write tests for this and all related features to make sure another related customization is not breaking this feature and also to save time on testing everything manually.
Behat tests are the right choice when you want to get the most from your tests with limited resources.
Also, we gladly invite everyone to stay in touch with Oro community, where you can ask more questions and discuss Behat usage with our team via our forums and Slack channel.