+0 with the mention that the process needs to be documented well on the Release Plan. As a RM I will need to know where and at what job / report to look at. Also our ci is very slow and the release process IMO gets harder and harder because of the long waiting times, sometimes waiting for 5 hours to have a build. I don't know how long this pipeline will take, but maybe we could improve somehow the machines the jobs are running. Thank you, Caty On Wed, Oct 3, 2018 at 1:34 PM Marius Dumitru Florea < mariusdumitru.florea@xwiki.com> wrote:
Sounds good. +1
Thanks, Marius
On Wed, Oct 3, 2018 at 11:56 AM Vincent Massol <vincent@massol.net> wrote:
Hi devs,
We started discussing a first global test coverage strategy in https://markmail.org/message/grphwta63pp5p4l7
I’d like to propose some updates and tuning now that we have a Clover Jenkins pipeline working (brainstormed with Simon):
Here’s the new proposal:
* We run the Clover Jenkins pipeline every night (between 11PM-8AM) * The pipeline sends an email whenever the new report has modules lowering the global TPC score compared to the baseline report (negative contribution per module) * The baseline report is the report generated just after each XS release. This means that we keep the same baseline during a XS release ** Technically it means that the pipeline will update the latest.txt file (which contains the clover report timestamp) when it notices a version change * We add a step in the Release Plan Template to have the report passing before we can release. * The RM is in charge of a release from day 1 to the release day (already the case), and is also in charge of making sure that the global coverage job failures get addressed before the release day so that we’re ready on the release day.
Options: * Make it easier and only fail the pipeline job when the global TPC value is lower than the baseline (vs failing whenever a module has negative contribution)
WDYT?
Thanks -Vincent