Edit in GitHubLog an issue

Reporting

The Functional Testing Framework provides two types of reporting:

When you run a test, the Functional Testing Framework copies all reporting artifacts to the dev/tests/acceptance/tests/_output subdirectory in the Adobe Commerce or Magento Open Source root directory. The directory contains:

  • allure-results/ that is a directory generated and served by the Allure Framework.
  • failed that is a text file containing relative paths to failed tests after the last test run. The paths are relative to dev/tests/acceptance/.
  • .html and .png files that are screenshots of fails in HTML and PNG formats. To cleanup the _output/ directory, remove them manually.

The mftf tool logs output continuously to the dev/tests/acceptance/mftf.log file.

Command line

The Functional Testing Framework reports about its progress during test run when you run the mftf command-line tool with run:test or run:group commands.

The report can contain three main parts:

  • Pre-run checks:
    • Environment check, such as PHP warnings, etc.
    • XML test validation like deprecation warnings such as missing required components in XML tests.
  • Codeception report which is the progress report for each test.
  • Total results of the test run such as number of tests, assertions, and failures.

To manage the level of verbosity, use -v or --verbose flag in the mftf commands. To enable verbosity using the codecept commands, refer to the Codeception Console Commands.

The following sections demonstrate an example interpretation of a complete log separated into chunks.

Pre-run check report

First, the Functional Testing Framework returns DEPRECATION warnings alerting you that required test components are missing in XML test definitions.

Copied to your clipboard
DEPRECATION: Test AdminFilteringCategoryProductsUsingScopeSelectorTest is missing required annotations.{"testName":"AdminFilteringCategoryProductsUsingScopeSelectorTest","missingAnnotations":"stories"}
DEPRECATION: Test AdminAbleToShipPartiallyInvoicedItemsTest is missing required annotations.{"testName":"AdminAbleToShipPartiallyInvoicedItemsTest","missingAnnotations":"stories"}
DEPRECATION: Test AdminRemoveProductWeeeAttributeOptionTest is missing required annotations.{"testName":"AdminRemoveProductWeeeAttributeOptionTest","missingAnnotations":"stories"}
Generate Tests Command Run

Generate Tests Command Run indicates that test generation is finished and tests are able to be executed.

Test execution report

A test execution report is generated by Codeception and includes configuration information, scenario execution steps, and PASSED/FAIL verdict after each test completion.

General information

The general information can be useful for Functional Testing Framework contributors, but can be ignored by a test writer.

Let's consider the general part of the following test execution report:

Copied to your clipboard
Generate Tests Command Run
Codeception PHP Testing Framework v4.1.4
Powered by PHPUnit 9.1.3 by Sebastian Bergmann and contributors.
Running with seed: 


Magento\FunctionalTestingFramework.functional Tests (2) ------------------------
Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer

After the test generation command (mentioned in the previous section), the Functional Testing Framework delegates control to the vendor/codeception tool, which is the Codeception PHP Testing Framework of version 4.1.4 that uses PHPUnit of version 9.1.3.

The tool runs 2 Tests using the configuration defined in the functional suite under the Magento\FunctionalTestingFramework namespace. The corresponding configuration file is acceptance/tests/functional.suite.yml. It enables Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Module\MagentoSequence, \Magento\FunctionalTestingFramework\Module\MagentoAssert, \Magento\FunctionalTestingFramework\Module\MagentoActionProxies, Asserts, \Magento\FunctionalTestingFramework\Helper\HelperContainer,..

Passed tests

The next chunk of the log reports about test execution of the first test:

Copied to your clipboard
AdminLoginSuccessfulTestCest: Admin login successful test
Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginSuccessfulTestCest:AdminLoginSuccessfulTest
Test: tests/functional/Magento/_generated/default/AdminLoginSuccessfulTestCest.php:AdminLoginSuccessfulTest
Scenario --
[loginAsAdmin] AdminLoginActionGroup
  [navigateToAdmin] am on page "/admin/admin"
  [fillUsername] fill field "#username","admin"
  [fillPassword] fill field "#login","123123q"
  [clickLogin] click ".actions .action-primary"
  [clickLoginWaitForPageLoad] wait for page load 30
  [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true
  [closeAdminNotification] close admin notification 
[assertLoggedIn] AssertAdminSuccessLoginActionGroup
  [waitForAdminAccountTextVisible] wait for element visible ".page-header .admin-user-account-text",60
  [assertAdminAccountTextElement] see element ".page-header .admin-user-account-text"
[logoutFromAdmin] AdminLogoutActionGroup
  [amOnLogoutPage] am on page "/admin/admin/auth/logout/"
 PASSED

The running test is AdminLoginSuccessfulTestCest, which is Admin login successful test (this text is generated from the test name but with the Cest part excluded). Its test signature is Magento\AcceptanceTest\_default\Backend\AdminLoginSuccessfulTestCest:AdminLoginSuccessfulTest that matches a className:methodName format using namespaces. A path to the corresponding Test is tests/functional/Magento/_generated/default/AdminLoginSuccessfulTestCest.php:AdminLoginSuccessfulTest (relative to the acceptance/ directory).

Scenario lists the tests steps as they run during test execution, ending with the successful test verdict PASSED. It means that all test steps were processed as expected.

Failed tests

The second test fails with the following report:

Copied to your clipboard
AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test
Signature: Magento\AcceptanceTest\_default\Backend\AdminMenuNavigationWithSecretKeysTestCest:AdminMenuNavigationWithSecretKeysTest
Test: tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest
Scenario --
[enableUrlSecretKeys] magento cli "config:set admin/security/use_form_key 1",60
Value was saved.


[cleanInvalidatedCaches1] magento cli "cache:clean config full_page",60
Cleaned cache types:
config
full_page


[loginAsAdmin] AdminLoginActionGroup
  [navigateToAdmin] am on page "/admin/admin"
  [fillUsername] fill field "#username","admin"
  [fillPassword] fill field "#login","123123q"
  [clickLogin] click ".actions .action-primary"
  [clickLoginWaitForPageLoad] wait for page load 30
  [clickDontAllowButtonIfVisible] conditional click ".modal-popup .action-secondary",".modal-popup .action-secondary",true
  [closeAdminNotification] close admin notification 
[clickStoresMenuOption1] click "#menu-magento-backend-stores"
[waitForStoresMenu1] wait for loading mask to disappear 
[clickStoresConfigurationMenuOption1] click "#nav li[data-ui-id='menu-magento-config-system-config']"
[waitForConfigurationPageLoad1] wait for page load 60
[seeCurrentUrlMatchesConfigPath1] see current url matches "~\/admin\/system_config\/~"
[clickCatalogMenuOption] click "#something"
[saveScreenshot] save screenshot 
[disableUrlSecretKeys] magento cli "config:set admin/security/use_form_key 0",60
Value was saved.


[cleanInvalidatedCaches2] magento cli "cache:clean config full_page",60
Cleaned cache types:
config
full_page


[logout] AdminLogoutActionGroup
  [amOnPage] am on page "/admin/admin/auth/logout/"
 FAIL 
--------------------------------------------------------------------------------

The general test details and scenario has the same format as in the Passed test.

Copied to your clipboard
[clickCatalogMenuOption] click "#something"
[saveScreenshot] save screenshot

When a test step fails, the Functional Testing Framework always saves a screenshot of the web page with the failing state immediately after the failure occurs. [saveScreenshot] save screenshot follows the failing test step [clickCatalogMenuOption] click "#something" in our case.

A screenshot of the fail goes at the acceptance/tests/_output directory in both PNG and HTML formats:

  • Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.html
  • Magento.AcceptanceTest._default.Backend.AdminMenuNavigationWithSecretKeysTestCest.AdminMenuNavigationWithSecretKeysTest.fail.png

The filename encodes:

  • Magento namespace
  • with the AcceptanceTest test type
  • generated as a part of the _default suite
  • defined at the Magento_Backend module
  • implemented in the AdminMenuNavigationWithSecretKeysTestCest PHP class
  • with the AdminMenuNavigationWithSecretKeysTest test name
  • and execution status fail

Actions after saveScreenshot are run as a part of the after hook of the test.

Test result report

After the Functional Testing Framework completes test execution, it generates a general report about test results along with detailed information about each fail.

Copied to your clipboard
Time: 02:07.534, Memory: 150.50 MB


There was 1 failure:
---------

MFTF reports that the test run took 02:07.534 using 150.50 MB of system RAM. And, finally, that there was 1 failure.

Next, the report provides details about the test failure.

Copied to your clipboard
---------
1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test
 Test  tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest
 Step  Click "#something"
 Fail  CSS or XPath element with '#something' was not found.


Scenario Steps:


 27. // Exiting Action Group [logout] AdminLogoutActionGroup
 26. $I->amOnPage("/admin/admin/auth/logout/") at tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:55
 25. // Entering Action Group [logout] AdminLogoutActionGroup
 24. // Cleaned cache types:
config
full_page


 23. $I->magentoCLI("cache:clean config full_page",60) at tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:52
 22. // Value was saved.

  • 1) AdminMenuNavigationWithSecretKeysTestCest: Admin menu navigation with secret keys test - the failed Codeception test is AdminMenuNavigationWithSecretKeysTestCest. It references to the PHP class that implemented the failed test.

  • Test tests/functional/Magento/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php:AdminMenuNavigationWithSecretKeysTest - the test is implemented in the AdminMenuNavigationWithSecretKeysTest test method of the tests/functional/Magento/FunctionalTest/_generated/default/AdminMenuNavigationWithSecretKeysTestCest.php file under <magento root>/dev/tests/acceptance/. It matches the corresponding test defined in XML that is AdminMenuNavigationWithSecretKeysTest defined in <test name="AdminMenuNavigationWithSecretKeysTest">...</test>

  • Step Click "#something" - the failing test step is the click action with the #something selector. It would correspond the <click selector="#something" ... /> test step in the XML defined tests.

Finally, the report finishes with fairly self-descriptive lines.

Copied to your clipboard
FAILURES!
Tests: 2, Assertions: 2, Failures: 1.

MFTF encountered failures due to the last test run, that included 2 tests with 2 assertions. 1 assertion fails.

Allure

Each time you run tests, the Functional Testing Framework appends an XML file with results at the tests/_output/allure-results/ directory.

The official Allure Test Report documentation is well-covered, so we'll list only the CLI commands that you would need for your day-to-day work.

To generate the HTML Allure report in a temporary folder and open the report in your default web browser:

Copied to your clipboard
allure serve dev/tests/acceptance/tests/_output/allure-results/

To generate a report to the allure-report/ at the current directory:

Copied to your clipboard
allure generate dev/tests/acceptance/tests/_output/allure-result

To generate a report to a particular directory, use the -o option:

Copied to your clipboard
allure generate dev/tests/acceptance/tests/_output/allure-result -o dev/tests/acceptance/tests/_output/allure-report

To launch the generated report in a web browser:

Copied to your clipboard
allure open dev/tests/acceptance/tests/_output/allure-report
Copied to your clipboard
allure generate dev/tests/acceptance/tests/_output/allure-result

Example of file structure after the command run:

Copied to your clipboard
magento2
├── allure-report
├── app
├── bin
├── dev
├── ...

And if you run the open command with no arguments while you are in the same directory (magento2/):

Copied to your clipboard
allure open

Allure would attempt to open a generated report at the magento2/allure-report/ directory.

To clean up existing reports before generation (for example after getting new results), use the --clean flag:

Copied to your clipboard
allure generate dev/tests/acceptance/tests/_output/allure-result --clean

Refer to the Reporting section for more Allure CLI details.

Jeff MatthewsDima Shevtsov
Last updated 7/10/2024
Was this helpful?

Community

Support

Adobe Developer

  • Privacy
  • Terms of Use
  • Do not sell or share my personal information
  • AdChoices
Copyright © 2024 Adobe. All rights reserved.