OpenProject uses automated tests throughout the stack. Tests that are executed in the browser (angular frontend, rspec system tests) require to have Chrome installed.
You will likely start working with the OpenProject test suite through our continuous testing setup at [Github Actions](https://github.com/opf/openproject/actions). All pull requests and commits to the core repository will be tested by Github Actions.
You will likely start working with the OpenProject test suite through our continuous testing setup at [GitHub Actions](https://github.com/opf/openproject/actions). All pull requests and commits to the core repository will be tested by GitHub Actions.
# Continuous testing with Github Actions
# Continuous testing with GitHub Actions
As part of the [development flow at OpenProject](../../development/#branching-model-and-development-flow), proposed changes to the core application will be made through a GitHub pull request and the entire test suite is automatically evaluated on Github Actions. You will see the results as a status on your pull request. Successful test suite runs are one requirement to see your changes merged.
As part of the [development flow at OpenProject](../../development/#branching-model-and-development-flow), proposed changes to the core application will be made through a GitHub pull request and the entire test suite is automatically evaluated on GitHub Actions. You will see the results as a status on your pull request.
Successful test suite runs are one requirement to see your changes merged.
## List failures
A failing status will look like the following on your pull request. You may need to click *Show all checks* to expand all checks to see the details link.
@ -16,9 +21,11 @@ A failing status will look like the following on your pull request. You may need
Here you'll see that the *Github Actions* check has reported an error, which likely means that your pull request contains errors. It might also result from a temporary error running the test suite, or from a test that was broken in the `dev` branch.
Here you'll see that the *GitHub Actions* check has reported an error, which likely means that your pull request contains errors. It might also result from a [temporary error running the test suite](#tests-failing-on-github-actions-ci-and-passing-locally), or from a test that was broken in the `dev` branch.
If you expand the view by clicking on details, you will see the individual *jobs* that Github executes. The test suite is run in parallel to save time. The overall run time of the test suite is around *15 minutes* on Github. Due to parallel test runs and beefier custom worker machines, the run time is significantly lower than on our previous test CI.
The test suite is [run in parallel](#parallel-testing) to save time. The overall run time of the test suite is around *15 minutes* on GitHub.
Click on the Details link to see the individual *jobs* that GitHub executes.
[Here's a link to an exemplary failed test run on GitHub](https://github.com/opf/openproject/pull/9355/checks?check_run_id=2730782867). In this case, one of the feature jobs has reported an error.
@ -26,7 +33,7 @@ If you expand the view by clicking on details, you will see the individual *job
You can click on each job and each step to show the [log output for this job](https://github.com/opf/openproject/pull/9355/checks?check_run_id=2730782867). It will contain more information about how many tests failed and will also temporarily provide a screenshot of the browser during the occurrence of the test failure (only if a browser was involved in testing).
Click on each job and each step to show the [log output for this job](https://github.com/opf/openproject/pull/9355/checks?check_run_id=2730782867). It will contain more information about how many tests failed and will also temporarily provide a screenshot of the browser during the occurrence of the test failure (only if a browser was involved in testing).
In our example, multiple tests are reported as failing:
```
@ -37,17 +44,20 @@ rspec ./spec/features/work_packages/timeline/timeline_navigation_spec.rb:193 # W
rspec ./spec/features/work_packages/timeline/timeline_navigation_spec.rb:317 # Work package timeline navigation when table is grouped shows milestone icons on collapsed project group rows but not on expanded ones
You can now run this test locally to try and reproduce the failure. How to do this depends on the kind of job that failed.
Once you know which tests are failing, run them locally to try and reproduce the failures. Having reproducible failures locally is the first step to diagnose and fix them.
How to do this depends on the kind of job that failed.
**Errors in the npm group**
If there is an error in the npm group, you likely have broken an existing Angular component spec or added an invalid new one. Please see the [Frontend tests section](#frontend-tests) on how to run them.
An error in the *npm* group means you likely have broken an existing Angular component spec or added an invalid new one. Please see the [Frontend tests](#frontend-tests) section on how to run them.
@ -69,7 +79,7 @@ For the `legacy specs` job, please [see the section on running legacy specs](#le
**Helper to extract all failing tests**
There is a small ruby script that will parse the logs of a Github Actions run and output all `rspec` tests that failed for you to run in one command.
There is a small ruby script that will parse the logs of a GitHub Actions run and output all `rspec` tests that failed for you to run in one command.
```
./script/github_pr_errors
@ -79,11 +89,40 @@ Note that it will output legacy specs and specs together, which need to be run s
### Skipping test execution on Github Actions CI
## Tests failing on GitHub Actions CI and passing locally
Some tests can fail on GitHub actions CI, and pass locally which makes them harder to reproduce, diagnose, and fix.
Possible reasons are:
* Different configuration between CI environment and local environment
* GitHub actions run with `CI=true` environment variable. This setting will eager load the app before running tests. As some classes may monkey patch parts of the code, the behavior can be different when the app is fully loaded.
* Try running the tests with `CI=true`.
* OpenProject configuration difference
* Try changing or disabling any environment variables prefixed with `OPENPROJECT_` in your environment or `.env` files.
* Try changing or removing `config/configuration.yml` settings under the `test:` key.
* Missing executables
* Source control management tests may need `svnadmin` or `git` to execute properly.
* LDAP tests may need `java` to spin up a LDAP server instance.
* Different test execution order
* Parts of the OpenProject code are using memoization and caching for performance, and some tests can do weird things like prepending a module or other meta programming. Without proper clean up of the global state, subsequent tests may fail. It can go unnoticed depending on the test execution order.
* RSpec tests order is different on each run. The order is defined by the random seed which can be set with `--seed` option. When running rspec, the random seed is displayed like this: `Randomized with seed 18352`.
* Try running tests locally with the same random seed as the one used on CI.
* Once you determined that the failure is order dependant, use [`--bisect`](https://relishapp.com/rspec/rspec-core/docs/command-line/bisect) to isolate the minimal set of examples that reproduce the same failures.
* Faster / slower machine and race conditions
* Some system tests using browser and performing ajax requests may not be synchronized with the test execution: the test is testing something that has not happened yet. Sometimes the ajax runs at the right time and the test passes, sometimes it runs at the wrong time and the test fails.
* Use `script/bulk_run_rspec` to run the same test multiple times. If it has both failing and passing results, it means it is a flickering test.
* To help diagnose why a system test is failing:
* Browser screenshots are created for failing system tests involving a browser. You can find them in the job log output.
* Try running with `OPENPROJECT_TESTING_NO_HEADLESS=1` to view what the browser is doing. Use `OPENPROJECT_TESTING_AUTO_DEVTOOLS=1` to have DevTools opened so that you can use `debugger` statements in the js code.
## Skip test execution on GitHub Actions CI
Sometimes, you know you're pushing changes to a pull request that you now are work in progress or are known to break existing or new tests.
To avoid additional test executions, you can include `[skip ci]` in your commit message to ensure Github Actions are not being triggered and skips your build. Please note that a successful merge of your pull request will require a green CI build.
To avoid additional test executions, you can include `[skip ci]` in your commit message to ensure GitHub Actions are not being triggered and skips your build. Please note that a successful merge of your pull request will require a green CI build.
@ -164,13 +203,13 @@ Alternatively, when in the `frontend/` folder, you can also use the watch mode o
## Unit tests
After following the prerequisites, you can simply use the following command to run individual specs:
After following the prerequisites, use the following command to run individual specs:
We use Capybara and Selenium for system tests, which are often also called as *rspec feature specs*. They are automatically executed with an actual browser when `js: true` is set.
System tests are also called *rspec feature specs* and use [Capybara](https://rubydoc.info/github/teamcapybara/capybara/master) and [Selenium](https://www.selenium.dev/documentation/webdriver/) to run. They are automatically executed with an actual browser when `js: true` is set.
System tests are located in `spec/features`. Use the following command to run individual test:
@ -204,7 +251,7 @@ The tests will generally run a lot slower due to the whole application being run
You can also run *all* feature specs locally with this command. This is not recommended due to the required execution time. Instead, prefer to select individual tests that you would like to test and let Github Actions CI test the entire suite.
You can also run *all* feature specs locally with this command. This is not recommended due to the required execution time. Instead, prefer to select individual tests that you would like to test and let GitHub Actions CI test the entire suite.
@ -294,9 +341,9 @@ You can fix this either by accessing a page locally (if the rails server is runn
You can run the specs with the following commands:
* `bundle exec rake spec` Run all core specs and feature tests. Again ensure that the Angular CLI is running for these to work. This will take a long time locally, and it is not recommend to run the entire suite locally. Instead, wait for the test suite run to be performed on Github Actions CI as part of your pull request.
* `bundle exec rake spec` Run all core specs and feature tests. Again ensure that the Angular CLI is running for these to work. This will take a long time locally, and it is not recommend to run the entire suite locally. Instead, wait for the test suite run to be performed on GitHub Actions CI as part of your pull request.
* `SPEC_OPTS="--seed 12935" bundle exec rake spec` Run the core specs with the seed 12935. Use this to control in what order the tests are run to identify order-dependent failures. You will find the seed that Github Actions CI used in their log output.
* `SPEC_OPTS="--seed 12935" bundle exec rake spec` Run the core specs with the seed 12935. Use this to control in what order the tests are run to identify order-dependent failures. You will find the seed that GitHub Actions CI used in their log output.
@ -18,7 +18,7 @@ keywords: date picker start finish dates change modify update relations work pac
You can change the start and finish dates of work packages by opening the date picker.
<imgsrc="standard-date-picker.png"alt="The standard date picker"style="zoom:50%;"/>
1. [Manual scheduling](#scheduling-mode) checkbox
2. Start and finish date input fields
@ -52,7 +52,7 @@ To confirm the selected dates, click on the **Save** button. The green message o
To clear a date field, simply click on the **clear icon** ("×") icon when the field is in focus.
<imgsrc="clear-date-field.png"alt="A focused date field has an X to clear it"style="zoom:50%;"/>
Date changes are documented in the work package [Activity](../../activity/).
@ -66,20 +66,20 @@ Date changes are documented in the work package [Activity](../../activity/).
If you want the work package to start and end on the same date, simply click the same date twice.
<imgsrc="one-day-event.png"alt="Date picker with the same start and finish dates"style="zoom:25%;"/>
Work packages with *only* a start date or only a finish date are automatically considered as one-day events.
Certain work package types (such as Milestones) can only span one day and thus have only one date field:
<imgsrc="milestone-datepicker.png"alt="Milestones have datpickers with a single date field"style="zoom:25%;"/>
## Scheduling mode
### Manual scheduling
<imgsrc="manual-scheduling-checkbox.png"alt="Datepicker with a checkbox to enable manual scheduling"style="zoom:50%;"/>
Checking "Manual scheduling" enables [manual scheduling mode](../../gantt-chart/scheduling/#manual-scheduling-mode) for the work package. This ignores work package relations such that you're able to select any start and end dates, even ones that would normally be restricted due to follows/proceeds relationships.
@ -97,7 +97,7 @@ There are four possible banners:
#### Automatically scheduled
<imgsrc="banner-automatically-scheduled.png"alt="A blue banner informing the user that the worked package is automatically scheduled"style="zoom:25%;"/>
This information banner is displayed when the dates of the current work package are derived from existing relations and can therefore not be modified. This is the case of parent work packages whose start and finish dates are derived from the earliest and latest child work packages respectively.
@ -105,13 +105,13 @@ It is nevertheless possible to input specific start and finish dates for such wo
### Available start date and finish dates limited by relations.
<imgsrc="banner-dates-limited.png"alt="A blue banner informing the user that certain dates are limited by relations"style="zoom:25%;"/>
This information banner is displayed when certain date ranges are disabled due to constraints imposed by follows/preceeds relations. For example, a work packages following another one cannot be scheduled before the finish date of the the preceeding one.
### Manually scheduled, relations ignored
<imgsrc="banner-manual-relations-ignored.png"alt="An orange banner warning the user that manual scheduling will ignore existing relations"style="zoom:25%;"/>
This warning banner is displayed when you enable manual scheduling on work packages that have follows/preceeds or parent/child relations, which are now ignored and no longer limit which dates can be selected for this work package. This also means that changes a manually scheduled work package no longer affect others, despite those exisiting relations.
@ -119,7 +119,7 @@ This banner is _not_ displayed when manual scheduling is enabled on work package
### Other work packages affected
<imgsrc="banner-other-workpackages-affected.png"alt="An orange banner warning the user that changing these dates will affect other the dates of work packages"style="zoom:25%;"/>
This warning banner is displayed on work packages whose start and end dates affect those of other related work packages. For example, when changing the finish date will consequently change the start date of another (following) work package and/or the finish date of a parent.
@ -127,7 +127,7 @@ This warning banner is displayed on work packages whose start and end dates affe
### Show Relations button
<imgsrc="banner-show-relations-button.png"alt="The Show Relations button in an information or warning banner"style="zoom:50%;"/>
The information and warning banners also feature a **Show Relations** button. Clicking on this will open a new tab that displays work packages with direct relations to the current work package in [Gantt view](../../gantt-chart), in hierarchy mode.
Oliver Günther
2 years ago
