[xwiki-devs] [Proposal] Improve our Test Coverage strategy (TAKE 3)
Hi devs, I remember that we talked about this new global test coverage strategy but I don’t recall where and I couldn’t find an email about it. So I’m reposting the strategy that I’ve just finished implementing. * Every day the Clover job runs: https://ci.xwiki.org/view/Tools/job/Clover/ * It generates a global coverage report and compares it automatically with the first report generated for the current version * If the global coverage is reduced, then: ** an email is sent to notifications@xwiki.org containing the report ** the job is marked as FAILING ** A failing badge is added to the job history ** A red large text is added to the job page and a link to the report is sent * Developers MUST fix the global coverage before we can release a version, so the it means the global coverage must not be reduced during a version. * The RM should check the CI (already part of his chores) and the “Recommended Failing” view on ci.xwiki.org must not have errors or failures (so that includes pitest job and clover job). Any comments? Does that seem ok to you? If you confirm it’s ok, I’ll document it on https://dev.xwiki.org/xwiki/bin/view/Community/Testing/TestCoverage/ Note: A few minutes ago an email has been sent since the global coverage has been reduced compared to yesterday. See http://maven.xwiki.org/site/clover/20190321/XWikiReport-20190320-0128-201903... So we need to fix it now. Thanks -Vincent
+1 On Thu, Mar 21, 2019 at 5:41 PM Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I remember that we talked about this new global test coverage strategy but I don’t recall where and I couldn’t find an email about it. So I’m reposting the strategy that I’ve just finished implementing.
* Every day the Clover job runs: https://ci.xwiki.org/view/Tools/job/Clover/ * It generates a global coverage report and compares it automatically with the first report generated for the current version * If the global coverage is reduced, then: ** an email is sent to notifications@xwiki.org containing the report ** the job is marked as FAILING ** A failing badge is added to the job history ** A red large text is added to the job page and a link to the report is sent * Developers MUST fix the global coverage before we can release a version, so the it means the global coverage must not be reduced during a version. * The RM should check the CI (already part of his chores) and the “Recommended Failing” view on ci.xwiki.org must not have errors or failures (so that includes pitest job and clover job).
Any comments? Does that seem ok to you?
If you confirm it’s ok, I’ll document it on https://dev.xwiki.org/xwiki/bin/view/Community/Testing/TestCoverage/
Note: A few minutes ago an email has been sent since the global coverage has been reduced compared to yesterday.
See http://maven.xwiki.org/site/clover/20190321/XWikiReport-20190320-0128-201903...
So we need to fix it now.
Thanks -Vincent
-- Thomas Mortagne
+1 Thanks, Marius On Thu, Mar 21, 2019 at 7:03 PM Thomas Mortagne <thomas.mortagne@xwiki.com> wrote:
+1
On Thu, Mar 21, 2019 at 5:41 PM Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I remember that we talked about this new global test coverage strategy
but I don’t recall where and I couldn’t find an email about it. So I’m reposting the strategy that I’ve just finished implementing.
* Every day the Clover job runs:
https://ci.xwiki.org/view/Tools/job/Clover/
* It generates a global coverage report and compares it automatically with the first report generated for the current version * If the global coverage is reduced, then: ** an email is sent to notifications@xwiki.org containing the report ** the job is marked as FAILING ** A failing badge is added to the job history ** A red large text is added to the job page and a link to the report is sent * Developers MUST fix the global coverage before we can release a version, so the it means the global coverage must not be reduced during a version. * The RM should check the CI (already part of his chores) and the “Recommended Failing” view on ci.xwiki.org must not have errors or failures (so that includes pitest job and clover job).
Any comments? Does that seem ok to you?
If you confirm it’s ok, I’ll document it on https://dev.xwiki.org/xwiki/bin/view/Community/Testing/TestCoverage/
Note: A few minutes ago an email has been sent since the global coverage has been reduced compared to yesterday.
See http://maven.xwiki.org/site/clover/20190321/XWikiReport-20190320-0128-201903...
So we need to fix it now.
Thanks -Vincent
-- Thomas Mortagne
participants (3)
-
Marius Dumitru Florea -
Thomas Mortagne -
Vincent Massol