[xwiki-devs] XAR Unit testing Performance analysis + Guidelines
FWIW I’ve done a quick analysis of the speed of executing a XAR unit test (see http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HXARTesting). Globally it takes 3ms for the first test on my machine. With a profiler the time is roughly doubled (6ms) and is spread like this (for stuff that takes the most time): * loadPage(): 1.1s ** TIKA init : 800ms (initialized in XWikiAttachment) * getRenderedContent(): 1.2s ** org.apache.xpath.jaxp.XPathImpl.evaluate(): 260ms ** javax.validation.Validation.buildDefaultValidatorFactory(): 260ms * init stuff: ** org.slf4j.LoggerFactory.getLogger(String): 804ms ** mocking: 288ms ** component registration: 260ms My only ideas for speeding slightly the execution time would be to ensure that TIKA is not initialized for the test until it is required. And maybe check why SLF4J takes so long and find a way to make it run faster (maybe use a noop logger). So those 2 items could win us a max of 1.6s out of 6s, so that’s 800ms out of 3s. Which would lower our exec time for a single test to close to 2 seconds. But I don’t see how to go below that value. Conclusion: ========== - IMO we shouldn’t call those tests unit tests but integration tests (that’s what they are in practice) - we should run them in the integration-tests profile to not slow down the main build too much - they’re still way faster than writing a functional tests (more than 1 minute to package and run - that’s 20 times slower at best), so when possible this should be favored when we want to validate some specific behavior of a wiki page or ensure that a regression won’t happen again WDYT? If we’re ok I’ll make the changes (integration-tests profile + rename to integration tests). Thanks -Vincent
On Fri, Sep 2, 2016 at 11:00 AM, Vincent Massol <vincent@massol.net> wrote:
FWIW I’ve done a quick analysis of the speed of executing a XAR unit test (see http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HXARTesting).
Globally it takes 3ms for the first test on my machine.
3 seconds
With a profiler the time is roughly doubled (6ms) and is spread like this (for stuff that takes the most time):
6 seconds
* loadPage(): 1.1s ** TIKA init : 800ms (initialized in XWikiAttachment)
* getRenderedContent(): 1.2s ** org.apache.xpath.jaxp.XPathImpl.evaluate(): 260ms ** javax.validation.Validation.buildDefaultValidatorFactory(): 260ms
* init stuff: ** org.slf4j.LoggerFactory.getLogger(String): 804ms ** mocking: 288ms ** component registration: 260ms
My only ideas for speeding slightly the execution time would be to ensure that TIKA is not initialized for the test until it is required. And maybe check why SLF4J takes so long and find a way to make it run faster (maybe use a noop logger).
So those 2 items could win us a max of 1.6s out of 6s, so that’s 800ms out of 3s. Which would lower our exec time for a single test to close to 2 seconds.
But I don’t see how to go below that value.
Conclusion: ==========
- IMO we shouldn’t call those tests unit tests but integration tests (that’s what they are in practice) - we should run them in the integration-tests profile to not slow down the main build too much - they’re still way faster than writing a functional tests (more than 1 minute to package and run - that’s 20 times slower at best), so when possible this should be favored when we want to validate some specific behavior of a wiki page or ensure that a regression won’t happen again
WDYT?
If we’re ok I’ll make the changes (integration-tests profile + rename to integration tests).
+1. This reminded me that I should write XAR integration test for the document tree (for the page that generates the JSON for the tree). Thanks, Marius
Thanks -Vincent
_______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
On Fri, Sep 2, 2016 at 10:00 AM, Vincent Massol <vincent@massol.net> wrote:
FWIW I’ve done a quick analysis of the speed of executing a XAR unit test (see http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HXARTesting).
Globally it takes 3ms for the first test on my machine.
With a profiler the time is roughly doubled (6ms) and is spread like this (for stuff that takes the most time):
* loadPage(): 1.1s ** TIKA init : 800ms (initialized in XWikiAttachment)
* getRenderedContent(): 1.2s ** org.apache.xpath.jaxp.XPathImpl.evaluate(): 260ms ** javax.validation.Validation.buildDefaultValidatorFactory(): 260ms
* init stuff: ** org.slf4j.LoggerFactory.getLogger(String): 804ms ** mocking: 288ms ** component registration: 260ms
Note that except for component registration and mocking (not sure about XPathImpl#evaluate), I think most of the others are about static stuff being initialized the first time so following tests in the same run should be way faster than the first one.
My only ideas for speeding slightly the execution time would be to ensure that TIKA is not initialized for the test until it is required. And maybe check why SLF4J takes so long and find a way to make it run faster (maybe use a noop logger).
So those 2 items could win us a max of 1.6s out of 6s, so that’s 800ms out of 3s. Which would lower our exec time for a single test to close to 2 seconds.
But I don’t see how to go below that value.
Conclusion: ==========
- IMO we shouldn’t call those tests unit tests but integration tests (that’s what they are in practice) - we should run them in the integration-tests profile to not slow down the main build too much - they’re still way faster than writing a functional tests (more than 1 minute to package and run - that’s 20 times slower at best), so when possible this should be favored when we want to validate some specific behavior of a wiki page or ensure that a regression won’t happen again
WDYT?
If we’re ok I’ll make the changes (integration-tests profile + rename to integration tests).
Thanks -Vincent
_______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne
On 02 Sep 2016, at 10:49, Thomas Mortagne <thomas.mortagne@xwiki.com> wrote:
On Fri, Sep 2, 2016 at 10:00 AM, Vincent Massol <vincent@massol.net> wrote:
FWIW I’ve done a quick analysis of the speed of executing a XAR unit test (see http://dev.xwiki.org/xwiki/bin/view/Community/Testing#HXARTesting).
Globally it takes 3ms for the first test on my machine.
With a profiler the time is roughly doubled (6ms) and is spread like this (for stuff that takes the most time):
* loadPage(): 1.1s ** TIKA init : 800ms (initialized in XWikiAttachment)
* getRenderedContent(): 1.2s ** org.apache.xpath.jaxp.XPathImpl.evaluate(): 260ms ** javax.validation.Validation.buildDefaultValidatorFactory(): 260ms
* init stuff: ** org.slf4j.LoggerFactory.getLogger(String): 804ms ** mocking: 288ms ** component registration: 260ms
Note that except for component registration and mocking (not sure about XPathImpl#evaluate), I think most of the others are about static stuff being initialized the first time so following tests in the same run should be way faster than the first one.
Yep, this is why I mentioned "Globally it takes 3s for the first test on my machine.”. The following tests take about 100+ms. But ATM we don’t have that many tests per test class for XAR testing. Right now we have 3 test classes: - 1 test class has 1 test - 1 test class has 2 tests - 1 test class has 4 tests So indeed in total it’s about 9.4s, which leads to an average of 1.3s per test. I still think it’s too important and takes too long to consider them unit tests and they should be in the category of integration tests. WDYT? Thanks -Vincent
My only ideas for speeding slightly the execution time would be to ensure that TIKA is not initialized for the test until it is required. And maybe check why SLF4J takes so long and find a way to make it run faster (maybe use a noop logger).
So those 2 items could win us a max of 1.6s out of 6s, so that’s 800ms out of 3s. Which would lower our exec time for a single test to close to 2 seconds.
But I don’t see how to go below that value.
Conclusion: ==========
- IMO we shouldn’t call those tests unit tests but integration tests (that’s what they are in practice) - we should run them in the integration-tests profile to not slow down the main build too much - they’re still way faster than writing a functional tests (more than 1 minute to package and run - that’s 20 times slower at best), so when possible this should be favored when we want to validate some specific behavior of a wiki page or ensure that a regression won’t happen again
WDYT?
If we’re ok I’ll make the changes (integration-tests profile + rename to integration tests).
Thanks -Vincent
_______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
-- Thomas Mortagne _______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
participants (3)
-
Marius Dumitru Florea -
Thomas Mortagne -
Vincent Massol