LilyPond Contributor's Guide: 9.4 Regtest comparison

[ << Regression tests ]	[Top][Contents][Index][ ? ]			[ Programming work >> ]
[ < Compiling regression tests ]		[ Up : Regression tests ]	[ Pixel-based regtest comparison > ]

9.4 Regtest comparison

Before modified code is committed to master (via staging), a regression test comparison must be completed to ensure that the changes have not caused problems with previously working code. The comparison is made automatically upon compiling the regression test suite twice.

Before making changes to the code, establish a baseline for the comparison by checking out the current git master, going to the ‘$LILYPOND_GIT/build/’ directory and running:
```
make clean # whenever any files in mf/ are modified
make test-baseline
```
Make your changes, or apply the patch(es) to consider.
Check for unintentional changes to the regtests:
```
make check
```
After this has finished, a regression test comparison will be available (relative to the current ‘build/’ directory) at:
```
out/test-results/index.html
```
For each regression test that differs between the baseline and the changed code, a regression test entry will be displayed. Ideally, the only changes would be the changes that you were working on. If regressions are introduced, they must be fixed before committing the code.
If you are happy with the results, then skip to the final step.
If you want to continue programming, then make any additional code changes, and continue.
To rerun just the test cases that make check identified as differing from the baseline, run make check again. This updates the regression list at ‘out/test-results/index.html’.
When all regressions have been resolved, the output list will be empty.
After resolving regressions incrementally, run another full check to prove that your changes have not caused any skipped cases to regress:
```
make test-clean
make check
```
This set of regression tests is not all that needs to remain working. Before committing, make sure that you have successfully run make with your changes. Also, weigh the risk of your changes against the time that make doc will take, and run it if it is appropriate.

Advanced note: Once a test baseline has been established, there is no need to run it again unless git master changed. In other words, if you work with several branches and want to do regtests comparison for all of them, you can make test-baseline with git master, checkout some branch, make check it, then switch to another branch, make test-clean and make check it without doing make test-baseline again.

[ << Regression tests ]	[Top][Contents][Index][ ? ]			[ Programming work >> ]
[ < Compiling regression tests ]		[ Up : Regression tests ]	[ Pixel-based regtest comparison > ]

9.4 Regtest comparison

LilyPond — Contributor’s Guide v2.21.1 (development-branch).