Using suites

With an increasing number of tests, it is important to have a mechanism to organize and consolidate them for ease-of-use.

What is a suite?

A suite is a collection of tests that are intended to test specific behaviors of Magento. It may contain initialization and clean up steps common to the included test cases. It allows you to include, exclude and/or group tests with preconditions and post conditions. You can create a suite referencing tests, test groups and modules.

How is a suite defined?

A suite should be created under <magento2 root>/dev/tests/acceptance/tests/_suite if it has cross-module references. If a suite references only a single module, it should be created under <module>/Test/Mftf/Suite. The generated tests for each suite are grouped into their own directory under <magento2 root>/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/.

What is the format of a suite?

A suite is comprised of blocks:

<before> : executes precondition once per suite run.
<after> : executes postcondition once per suite run.
<include>: includes specific tests/groups/modules in the suite.
<exclude>: excludes specific tests/groups/modules from the suite.

<?xml version="1.0" encoding="UTF-8"?>

<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Suite/etc/suiteSchema.xsd">
    <suite name="">
        <before>
        </before>
        <after>
        </after>
        <include>
            <test name=""/>
            <group name=""/>
            <module name="" file=""/>
        </include>
        <exclude>
            <test name=""/>
            <group name=""/>
            <module name="" file=""/>
        </exclude>
    </suite>
</suites>
Copied to your clipboard
<?xml version="1.0" encoding="UTF-8"?>

<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Suite/etc/suiteSchema.xsd">
    <suite name="">
        <before>
        </before>
        <after>
        </after>
        <include>
            <test name=""/>
            <group name=""/>
            <module name="" file=""/>
        </include>
        <exclude>
            <test name=""/>
            <group name=""/>
            <module name="" file=""/>
        </exclude>
    </suite>
</suites>

Example

<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Suite/etc/suiteSchema.xsd">
    <suite name="WYSIWYGDisabledSuite">
        <before>
            <magentoCLI stepKey="disableWYSIWYG" command="config:set cms/wysiwyg/enabled disabled" />
        </before>
        <after>
            <magentoCLI stepKey="enableWYSIWYG" command="config:set cms/wysiwyg/enabled enabled" />
        </after>
        <include>
            <module name="Catalog"/>
        </include>
        <exclude>
            <test name="WYSIWYGIncompatibleTest"/>
        </exclude>
    </suite>
</suites>
Copied to your clipboard
<suites xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Suite/etc/suiteSchema.xsd">
    <suite name="WYSIWYGDisabledSuite">
        <before>
            <magentoCLI stepKey="disableWYSIWYG" command="config:set cms/wysiwyg/enabled disabled" />
        </before>
        <after>
            <magentoCLI stepKey="enableWYSIWYG" command="config:set cms/wysiwyg/enabled enabled" />
        </after>
        <include>
            <module name="Catalog"/>
        </include>
        <exclude>
            <test name="WYSIWYGIncompatibleTest"/>
        </exclude>
    </suite>
</suites>

This example declares a suite with name WYSIWYGDisabledSuite:

Disables WYSIWYG of the Adobe Commerce or Magento Open Source instance before running the tests.
Runs all tests from the Catalog module, except WYSIWYGIncompatibleTest
Returns the Adobe Commerce or Magento Open Source instance back to its original state, by enabling WYSIWYG at the end of testing.

Using suite commands

Generate all tests within a suite.

vendor/bin/mftf generate:suite <suiteName> [<suiteName>]
Copied to your clipboard
vendor/bin/mftf generate:suite <suiteName> [<suiteName>]

Run all tests within suite.

vendor/bin/mftf run:group <suiteName> [<suiteName>]
Copied to your clipboard
vendor/bin/mftf run:group <suiteName> [<suiteName>]

Generates any combination of suites and tests.

vendor/bin/mftf generate:tests --tests '{"tests":["testName1","testName2"],"suites":{"suite1":["suite_test1"],"suite2":null}}'
Copied to your clipboard
vendor/bin/mftf generate:tests --tests '{"tests":["testName1","testName2"],"suites":{"suite1":["suite_test1"],"suite2":null}}'

Run specific tests within a suite

If a test is referenced in a suite, it can be run in the suite's context with the run command. If a test is referenced in multiple suites, the run command will run the test multiple times in all contexts.

vendor/bin/mftf run:test <testName> [<testName>]
Copied to your clipboard
vendor/bin/mftf run:test <testName> [<testName>]

When to use suites?

Suites are a great way to organize tests which need the Adobe Commerce or Magento Open Source to be configured in a specific way as a pre-requisite. The conditions are executed once per suite which optimizes test execution time. If you wish to categorize tests solely based on functionality, use group tags instead.

Tips and tricks

Troubleshooting