[xwiki-devs] Status on Docker + CI + Clover - Outstanding tasks
Hi devs, I wanted to keep up to date on the list of stuff that we still need to do in the domain of Docker, CI & Clover, specifically on the domains I’ve been working on. All the items below are items I need to work on (but any help is welcome of course). 1) Clover pipeline job to modify to add docker-based functional tests: right now the global TPC is going to decrease because of MenuIT and MailIT. This needs to be fixed ASAP before we convert too many tests. 2) Also fix Clover pipeline job in general which hasn’t passed since the 22nd of October 3) Execute Docker-based tests in main pipeline job for the default config as otherwise we might have the platform job succeeding but it’ll miss several functional tests! Option 1: execute Browser outside of docker as option, Option 2: add more docker-based agents. For the moment, we might be able to run the docker tests on agent4. 4) Screenshot attachment on UI test failures to fix in pipeline job 5) Flicker recognition on tests to fix in pipeline job 6) Fix the ClosedChannelExcption/InterruptedException in our CI 7) Research running Jenkins agents in docker images and make our Docker test work (docker on docker). Once this works, set up 2 or 2 agents on a4. And then move all other agents to this. 8) Still need to apply the Clover strategy we agreed on (global TPC needs to not go down during releases). 9) Find strategy to move Docker image creation in platform build Let me know if you have questions. Thanks -Vincent PS: I didn’t list fixing tests (such and flicker fixing or WCAG test fixing) since this not infra and I wanted to limit this list to infra.
On 26 Nov 2018, at 00:44, Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I wanted to keep up to date on the list of stuff that we still need to do in the domain of Docker, CI & Clover, specifically on the domains I’ve been working on. All the items below are items I need to work on (but any help is welcome of course).
1) Clover pipeline job to modify to add docker-based functional tests: right now the global TPC is going to decrease because of MenuIT and MailIT. This needs to be fixed ASAP before we convert too many tests.
This should be good now. One more idea: 1.1) In the XWiki Clover report, compare the failing tests reported by Clover between old report (reference point) and the new one to gauge if the TPC computed is meaningful or not. 1.2) In the XWiki clover report, provide a diff list of executed tests to find out problems with some tests not executed which would lead to TPC being wrong.
2) Also fix Clover pipeline job in general which hasn’t passed since the 22nd of October
Just passed today. Need to monitor it. Future failures should be related to point 6) below. Note that the clover job is now executed on a4 which should make it a lot more stable. Thanks -Vincent
3) Execute Docker-based tests in main pipeline job for the default config as otherwise we might have the platform job succeeding but it’ll miss several functional tests! Option 1: execute Browser outside of docker as option, Option 2: add more docker-based agents. For the moment, we might be able to run the docker tests on agent4.
4) Screenshot attachment on UI test failures to fix in pipeline job
5) Flicker recognition on tests to fix in pipeline job
6) Fix the ClosedChannelExcption/InterruptedException in our CI
7) Research running Jenkins agents in docker images and make our Docker test work (docker on docker). Once this works, set up 2 or 2 agents on a4. And then move all other agents to this.
8) Still need to apply the Clover strategy we agreed on (global TPC needs to not go down during releases).
9) Find strategy to move Docker image creation in platform build
Let me know if you have questions.
Thanks -Vincent
PS: I didn’t list fixing tests (such and flicker fixing or WCAG test fixing) since this not infra and I wanted to limit this list to infra.
On 27 Nov 2018, at 18:19, Vincent Massol <vincent@massol.net> wrote:
On 26 Nov 2018, at 00:44, Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I wanted to keep up to date on the list of stuff that we still need to do in the domain of Docker, CI & Clover, specifically on the domains I’ve been working on. All the items below are items I need to work on (but any help is welcome of course).
1) Clover pipeline job to modify to add docker-based functional tests: right now the global TPC is going to decrease because of MenuIT and MailIT. This needs to be fixed ASAP before we convert too many tests.
This should be good now.
One more idea:
1.1) In the XWiki Clover report, compare the failing tests reported by Clover between old report (reference point) and the new one to gauge if the TPC computed is meaningful or not.
1.2) In the XWiki clover report, provide a diff list of executed tests to find out problems with some tests not executed which would lead to TPC being wrong.
I’ve almost finished to implement this but there’s a problem… For some reason I don’t understand Clover doesn’t report correctly failing tests: 1) A lot of tests are failing in the clover report but not in the build. For example: In Clover build logs: " [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in IdGeneratorTest [INFO] Running TransformationContextTest “ In clover.xml report: https://gist.github.com/vmassol/c8d3103147ddde3610d0c06bf44e37e1 ie 2 tests failing in IdGeneratorTest: testGenerateUniqueIdWhenInvalidEmptyPrefix & testGenerateUniqueIdWhenInvalidNonAlphaPrefix 2) Several tests failing in the Clover build logs but not in the clover report. For example: In Clover build logs: “ [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 139.958 s <<< FAILURE! - in org.xwiki.menu.test.ui.MenuIT [ERROR] org.xwiki.menu.test.ui.MenuIT Time elapsed: 139.958 s <<< ERROR! “ In clover.xml report: https://gist.github.com/vmassol/b5d3ba0120f08fa6e52e09157cad2a45 ie no failure at all… Thus it’s not possible to trust the clover.xml report… The only solution I can think of would be to compare either with the local Maven build logs or with the Jenkins test report logs. But that means we can’t go back in the past since we don’t have those logs for past clover reports. The good news is that when Clover says that tests failed, they actually didn't fail for a lot of them (However we would need to check if the coverage done by those tests is included but I think they are from what I see: IdGenerator.java has coveredelements="75” & elements=“84”. That could be from tests other than IdGeneratorTest though). Thanks -Vincent
2) Also fix Clover pipeline job in general which hasn’t passed since the 22nd of October
Just passed today. Need to monitor it. Future failures should be related to point 6) below. Note that the clover job is now executed on a4 which should make it a lot more stable.
Thanks -Vincent
3) Execute Docker-based tests in main pipeline job for the default config as otherwise we might have the platform job succeeding but it’ll miss several functional tests! Option 1: execute Browser outside of docker as option, Option 2: add more docker-based agents. For the moment, we might be able to run the docker tests on agent4.
4) Screenshot attachment on UI test failures to fix in pipeline job
5) Flicker recognition on tests to fix in pipeline job
6) Fix the ClosedChannelExcption/InterruptedException in our CI
7) Research running Jenkins agents in docker images and make our Docker test work (docker on docker). Once this works, set up 2 or 2 agents on a4. And then move all other agents to this.
8) Still need to apply the Clover strategy we agreed on (global TPC needs to not go down during releases).
9) Find strategy to move Docker image creation in platform build
Let me know if you have questions.
Thanks -Vincent
PS: I didn’t list fixing tests (such and flicker fixing or WCAG test fixing) since this not infra and I wanted to limit this list to infra.
On 2 Dec 2018, at 17:02, Vincent Massol <vincent@massol.net> wrote:
On 27 Nov 2018, at 18:19, Vincent Massol <vincent@massol.net> wrote:
On 26 Nov 2018, at 00:44, Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I wanted to keep up to date on the list of stuff that we still need to do in the domain of Docker, CI & Clover, specifically on the domains I’ve been working on. All the items below are items I need to work on (but any help is welcome of course).
1) Clover pipeline job to modify to add docker-based functional tests: right now the global TPC is going to decrease because of MenuIT and MailIT. This needs to be fixed ASAP before we convert too many tests.
This should be good now.
One more idea:
1.1) In the XWiki Clover report, compare the failing tests reported by Clover between old report (reference point) and the new one to gauge if the TPC computed is meaningful or not.
1.2) In the XWiki clover report, provide a diff list of executed tests to find out problems with some tests not executed which would lead to TPC being wrong.
I’ve almost finished to implement this but there’s a problem… For some reason I don’t understand Clover doesn’t report correctly failing tests:
1) A lot of tests are failing in the clover report but not in the build. For example:
In Clover build logs:
" [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in IdGeneratorTest [INFO] Running TransformationContextTest “
In clover.xml report: https://gist.github.com/vmassol/c8d3103147ddde3610d0c06bf44e37e1
ie 2 tests failing in IdGeneratorTest: testGenerateUniqueIdWhenInvalidEmptyPrefix & testGenerateUniqueIdWhenInvalidNonAlphaPrefix
2) Several tests failing in the Clover build logs but not in the clover report. For example:
In Clover build logs:
“ [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 139.958 s <<< FAILURE! - in org.xwiki.menu.test.ui.MenuIT [ERROR] org.xwiki.menu.test.ui.MenuIT Time elapsed: 139.958 s <<< ERROR! “
In clover.xml report: https://gist.github.com/vmassol/b5d3ba0120f08fa6e52e09157cad2a45
ie no failure at all…
Thus it’s not possible to trust the clover.xml report… The only solution I can think of would be to compare either with the local Maven build logs or with the Jenkins test report logs. But that means we can’t go back in the past since we don’t have those logs for past clover reports.
The good news is that when Clover says that tests failed, they actually didn't fail for a lot of them (However we would need to check if the coverage done by those tests is included but I think they are from what I see: IdGenerator.java has coveredelements="75” & elements=“84”. That could be from tests other than IdGeneratorTest though).
hmm found this in some forum: “ Regarding, your original question regarding failing tests. Yes, if test is failing Clover won't include its contribution to overall coverage stats. The reasoning behind this behavior is that since the test fails, its not trustworthy to say it provide test coverage (confidence) to your code. “ https://community.atlassian.com/t5/Clover-questions/clover-xml-does-not-cont... So maybe they’re not counted after all… Thanks -Vincent
Thanks -Vincent
2) Also fix Clover pipeline job in general which hasn’t passed since the 22nd of October
Just passed today. Need to monitor it. Future failures should be related to point 6) below. Note that the clover job is now executed on a4 which should make it a lot more stable.
Thanks -Vincent
3) Execute Docker-based tests in main pipeline job for the default config as otherwise we might have the platform job succeeding but it’ll miss several functional tests! Option 1: execute Browser outside of docker as option, Option 2: add more docker-based agents. For the moment, we might be able to run the docker tests on agent4.
4) Screenshot attachment on UI test failures to fix in pipeline job
5) Flicker recognition on tests to fix in pipeline job
6) Fix the ClosedChannelExcption/InterruptedException in our CI
7) Research running Jenkins agents in docker images and make our Docker test work (docker on docker). Once this works, set up 2 or 2 agents on a4. And then move all other agents to this.
8) Still need to apply the Clover strategy we agreed on (global TPC needs to not go down during releases).
9) Find strategy to move Docker image creation in platform build
Let me know if you have questions.
Thanks -Vincent
PS: I didn’t list fixing tests (such and flicker fixing or WCAG test fixing) since this not infra and I wanted to limit this list to infra.
On 2 Dec 2018, at 17:05, Vincent Massol <vincent@massol.net> wrote:
On 2 Dec 2018, at 17:02, Vincent Massol <vincent@massol.net> wrote:
On 27 Nov 2018, at 18:19, Vincent Massol <vincent@massol.net> wrote:
On 26 Nov 2018, at 00:44, Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I wanted to keep up to date on the list of stuff that we still need to do in the domain of Docker, CI & Clover, specifically on the domains I’ve been working on. All the items below are items I need to work on (but any help is welcome of course).
1) Clover pipeline job to modify to add docker-based functional tests: right now the global TPC is going to decrease because of MenuIT and MailIT. This needs to be fixed ASAP before we convert too many tests.
This should be good now.
One more idea:
1.1) In the XWiki Clover report, compare the failing tests reported by Clover between old report (reference point) and the new one to gauge if the TPC computed is meaningful or not.
1.2) In the XWiki clover report, provide a diff list of executed tests to find out problems with some tests not executed which would lead to TPC being wrong.
I’ve almost finished to implement this but there’s a problem… For some reason I don’t understand Clover doesn’t report correctly failing tests:
1) A lot of tests are failing in the clover report but not in the build. For example:
In Clover build logs:
" [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in IdGeneratorTest [INFO] Running TransformationContextTest “
In clover.xml report: https://gist.github.com/vmassol/c8d3103147ddde3610d0c06bf44e37e1
ie 2 tests failing in IdGeneratorTest: testGenerateUniqueIdWhenInvalidEmptyPrefix & testGenerateUniqueIdWhenInvalidNonAlphaPrefix
2) Several tests failing in the Clover build logs but not in the clover report. For example:
In Clover build logs:
“ [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 139.958 s <<< FAILURE! - in org.xwiki.menu.test.ui.MenuIT [ERROR] org.xwiki.menu.test.ui.MenuIT Time elapsed: 139.958 s <<< ERROR! “
In clover.xml report: https://gist.github.com/vmassol/b5d3ba0120f08fa6e52e09157cad2a45
ie no failure at all…
Thus it’s not possible to trust the clover.xml report… The only solution I can think of would be to compare either with the local Maven build logs or with the Jenkins test report logs. But that means we can’t go back in the past since we don’t have those logs for past clover reports.
The good news is that when Clover says that tests failed, they actually didn't fail for a lot of them (However we would need to check if the coverage done by those tests is included but I think they are from what I see: IdGenerator.java has coveredelements="75” & elements=“84”. That could be from tests other than IdGeneratorTest though).
hmm found this in some forum:
“ Regarding, your original question regarding failing tests. Yes, if test is failing Clover won't include its contribution to overall coverage stats. The reasoning behind this behavior is that since the test fails, its not trustworthy to say it provide test coverage (confidence) to your code. “
https://community.atlassian.com/t5/Clover-questions/clover-xml-does-not-cont...
So maybe they’re not counted after all…
haha there’s a way if we want to include them: “includeFailedTestCoverage”, see: http://openclover.org/doc/manual/4.2.0/ant--clover-check.html Thanks -Vincent
Thanks -Vincent
Thanks -Vincent
2) Also fix Clover pipeline job in general which hasn’t passed since the 22nd of October
Just passed today. Need to monitor it. Future failures should be related to point 6) below. Note that the clover job is now executed on a4 which should make it a lot more stable.
Thanks -Vincent
3) Execute Docker-based tests in main pipeline job for the default config as otherwise we might have the platform job succeeding but it’ll miss several functional tests! Option 1: execute Browser outside of docker as option, Option 2: add more docker-based agents. For the moment, we might be able to run the docker tests on agent4.
4) Screenshot attachment on UI test failures to fix in pipeline job
5) Flicker recognition on tests to fix in pipeline job
6) Fix the ClosedChannelExcption/InterruptedException in our CI
7) Research running Jenkins agents in docker images and make our Docker test work (docker on docker). Once this works, set up 2 or 2 agents on a4. And then move all other agents to this.
8) Still need to apply the Clover strategy we agreed on (global TPC needs to not go down during releases).
9) Find strategy to move Docker image creation in platform build
Let me know if you have questions.
Thanks -Vincent
PS: I didn’t list fixing tests (such and flicker fixing or WCAG test fixing) since this not infra and I wanted to limit this list to infra.
On 2 Dec 2018, at 17:05, Vincent Massol <vincent@massol.net> wrote:
On 2 Dec 2018, at 17:02, Vincent Massol <vincent@massol.net> wrote:
On 27 Nov 2018, at 18:19, Vincent Massol <vincent@massol.net> wrote:
On 26 Nov 2018, at 00:44, Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I wanted to keep up to date on the list of stuff that we still need to do in the domain of Docker, CI & Clover, specifically on the domains I’ve been working on. All the items below are items I need to work on (but any help is welcome of course).
1) Clover pipeline job to modify to add docker-based functional tests: right now the global TPC is going to decrease because of MenuIT and MailIT. This needs to be fixed ASAP before we convert too many tests.
This should be good now.
One more idea:
1.1) In the XWiki Clover report, compare the failing tests reported by Clover between old report (reference point) and the new one to gauge if the TPC computed is meaningful or not.
1.2) In the XWiki clover report, provide a diff list of executed tests to find out problems with some tests not executed which would lead to TPC being wrong.
I’ve almost finished to implement this but there’s a problem… For some reason I don’t understand Clover doesn’t report correctly failing tests:
1) A lot of tests are failing in the clover report but not in the build. For example:
In Clover build logs:
" [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in IdGeneratorTest [INFO] Running TransformationContextTest “
In clover.xml report: https://gist.github.com/vmassol/c8d3103147ddde3610d0c06bf44e37e1
ie 2 tests failing in IdGeneratorTest: testGenerateUniqueIdWhenInvalidEmptyPrefix & testGenerateUniqueIdWhenInvalidNonAlphaPrefix
2) Several tests failing in the Clover build logs but not in the clover report. For example:
In Clover build logs:
“ [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 139.958 s <<< FAILURE! - in org.xwiki.menu.test.ui.MenuIT [ERROR] org.xwiki.menu.test.ui.MenuIT Time elapsed: 139.958 s <<< ERROR! “
In clover.xml report: https://gist.github.com/vmassol/b5d3ba0120f08fa6e52e09157cad2a45
ie no failure at all…
Thus it’s not possible to trust the clover.xml report… The only solution I can think of would be to compare either with the local Maven build logs or with the Jenkins test report logs. But that means we can’t go back in the past since we don’t have those logs for past clover reports.
The good news is that when Clover says that tests failed, they actually didn't fail for a lot of them (However we would need to check if the coverage done by those tests is included but I think they are from what I see: IdGenerator.java has coveredelements="75” & elements=“84”. That could be from tests other than IdGeneratorTest though).
hmm found this in some forum:
“ Regarding, your original question regarding failing tests. Yes, if test is failing Clover won't include its contribution to overall coverage stats. The reasoning behind this behavior is that since the test fails, its not trustworthy to say it provide test coverage (confidence) to your code. “
https://community.atlassian.com/t5/Clover-questions/clover-xml-does-not-cont...
So maybe they’re not counted after all…
Ok seems it’s actually counted in the global report and TPC, see: https://stackoverflow.com/questions/41500243/does-clover-report-coverage-of-... " • global coverage - from entire test run, this will always contain code coverage from all tests (and not only tests, also from setUp methods etc) • per-test coverage - code coverage is tracked for every test individually “ Thanks -Vincent
Thanks -Vincent
Thanks -Vincent
2) Also fix Clover pipeline job in general which hasn’t passed since the 22nd of October
Just passed today. Need to monitor it. Future failures should be related to point 6) below. Note that the clover job is now executed on a4 which should make it a lot more stable.
Thanks -Vincent
3) Execute Docker-based tests in main pipeline job for the default config as otherwise we might have the platform job succeeding but it’ll miss several functional tests! Option 1: execute Browser outside of docker as option, Option 2: add more docker-based agents. For the moment, we might be able to run the docker tests on agent4.
4) Screenshot attachment on UI test failures to fix in pipeline job
5) Flicker recognition on tests to fix in pipeline job
6) Fix the ClosedChannelExcption/InterruptedException in our CI
7) Research running Jenkins agents in docker images and make our Docker test work (docker on docker). Once this works, set up 2 or 2 agents on a4. And then move all other agents to this.
8) Still need to apply the Clover strategy we agreed on (global TPC needs to not go down during releases).
9) Find strategy to move Docker image creation in platform build
Let me know if you have questions.
Thanks -Vincent
PS: I didn’t list fixing tests (such and flicker fixing or WCAG test fixing) since this not infra and I wanted to limit this list to infra.
On 2 Dec 2018, at 17:02, Vincent Massol <vincent@massol.net> wrote:
On 27 Nov 2018, at 18:19, Vincent Massol <vincent@massol.net> wrote:
On 26 Nov 2018, at 00:44, Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I wanted to keep up to date on the list of stuff that we still need to do in the domain of Docker, CI & Clover, specifically on the domains I’ve been working on. All the items below are items I need to work on (but any help is welcome of course).
1) Clover pipeline job to modify to add docker-based functional tests: right now the global TPC is going to decrease because of MenuIT and MailIT. This needs to be fixed ASAP before we convert too many tests.
This should be good now.
One more idea:
1.1) In the XWiki Clover report, compare the failing tests reported by Clover between old report (reference point) and the new one to gauge if the TPC computed is meaningful or not.
1.2) In the XWiki clover report, provide a diff list of executed tests to find out problems with some tests not executed which would lead to TPC being wrong.
I’ve almost finished to implement this but there’s a problem… For some reason I don’t understand Clover doesn’t report correctly failing tests:
1) A lot of tests are failing in the clover report but not in the build. For example:
In Clover build logs:
" [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.011 s - in IdGeneratorTest [INFO] Running TransformationContextTest “
In clover.xml report: https://gist.github.com/vmassol/c8d3103147ddde3610d0c06bf44e37e1
ie 2 tests failing in IdGeneratorTest: testGenerateUniqueIdWhenInvalidEmptyPrefix & testGenerateUniqueIdWhenInvalidNonAlphaPrefix
2) Several tests failing in the Clover build logs but not in the clover report. For example:
In Clover build logs:
“ [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 139.958 s <<< FAILURE! - in org.xwiki.menu.test.ui.MenuIT [ERROR] org.xwiki.menu.test.ui.MenuIT Time elapsed: 139.958 s <<< ERROR! “
In clover.xml report: https://gist.github.com/vmassol/b5d3ba0120f08fa6e52e09157cad2a45
ie no failure at all…
Thus it’s not possible to trust the clover.xml report… The only solution I can think of would be to compare either with the local Maven build logs or with the Jenkins test report logs. But that means we can’t go back in the past since we don’t have those logs for past clover reports.
The good news is that when Clover says that tests failed, they actually didn't fail for a lot of them (However we would need to check if the coverage done by those tests is included but I think they are from what I see: IdGenerator.java has coveredelements="75” & elements=“84”. That could be from tests other than IdGeneratorTest though).
3) Some tests fail because of Clover. For example IncludeMacroTest: https://gist.github.com/vmassol/30cb78322dc5a4a50ee5d4a41c383a79 We should be able to figure out the problem with his one. Thanks -Vincent
Thanks -Vincent
2) Also fix Clover pipeline job in general which hasn’t passed since the 22nd of October
Just passed today. Need to monitor it. Future failures should be related to point 6) below. Note that the clover job is now executed on a4 which should make it a lot more stable.
Thanks -Vincent
3) Execute Docker-based tests in main pipeline job for the default config as otherwise we might have the platform job succeeding but it’ll miss several functional tests! Option 1: execute Browser outside of docker as option, Option 2: add more docker-based agents. For the moment, we might be able to run the docker tests on agent4.
4) Screenshot attachment on UI test failures to fix in pipeline job
5) Flicker recognition on tests to fix in pipeline job
6) Fix the ClosedChannelExcption/InterruptedException in our CI
7) Research running Jenkins agents in docker images and make our Docker test work (docker on docker). Once this works, set up 2 or 2 agents on a4. And then move all other agents to this.
8) Still need to apply the Clover strategy we agreed on (global TPC needs to not go down during releases).
9) Find strategy to move Docker image creation in platform build
Let me know if you have questions.
Thanks -Vincent
PS: I didn’t list fixing tests (such and flicker fixing or WCAG test fixing) since this not infra and I wanted to limit this list to infra.
Status as of today below.
On 26 Nov 2018, at 00:44, Vincent Massol <vincent@massol.net> wrote:
Hi devs,
I wanted to keep up to date on the list of stuff that we still need to do in the domain of Docker, CI & Clover, specifically on the domains I’ve been working on. All the items below are items I need to work on (but any help is welcome of course).
1) Clover pipeline job to modify to add docker-based functional tests: right now the global TPC is going to decrease because of MenuIT and MailIT. This needs to be fixed ASAP before we convert too many tests.
Done
2) Also fix Clover pipeline job in general which hasn’t passed since the 22nd of October
Done
3) Execute Docker-based tests in main pipeline job for the default config as otherwise we might have the platform job succeeding but it’ll miss several functional tests! Option 1: execute Browser outside of docker as option, Option 2: add more docker-based agents. For the moment, we might be able to run the docker tests on agent4.
Not done yet, need to have step 7 work first and move all agents to docker.
4) Screenshot attachment on UI test failures to fix in pipeline job
Tried to fix it but not fully working yet.
5) Flicker recognition on tests to fix in pipeline job
Tried to fix it but not fully working yet.
6) Fix the ClosedChannelExcption/InterruptedException in our CI
I think we still have it from time to time but not sure.
7) Research running Jenkins agents in docker images and make our Docker test work (docker on docker). Once this works, set up 2 or 2 agents on a4. And then move all other agents to this.
Done!
8) Still need to apply the Clover strategy we agreed on (global TPC needs to not go down during releases).
First version done, I’ve proposed a new version that I now need to implement.
9) Find strategy to move Docker image creation in platform build
Not done yet. Thanks -Vincent
Let me know if you have questions.
Thanks -Vincent
PS: I didn’t list fixing tests (such and flicker fixing or WCAG test fixing) since this not infra and I wanted to limit this list to infra.
participants (1)
-
Vincent Massol