[xwiki-devs] Integration tests that needs Internet connectivity to pass
Hello devs, This is a small wondering I am having while reviewing selenium tests for XWiki Watch. Quoting my last remark on XWATCH-182 : I am still not sure whether or not tests that needs Internet connectivity are acceptable (such as the tests that add feeds from distant sources as Google Blog Search, etc.) I tend to think it is acceptable, as the only case one would try to run the tests offline, he would have to do it explicitly (with mvn -o), and thus would understand that a test method with a name like "testAddRSSFeedFromGoogleBlogSearch" could indeed fail. WDYT ? Regards, Jerome
On Jul 23, 2008, at 9:18 AM, Jerome Velociter wrote:
Hello devs,
This is a small wondering I am having while reviewing selenium tests for XWiki Watch. Quoting my last remark on XWATCH-182 :
I am still not sure whether or not tests that needs Internet connectivity are acceptable (such as the tests that add feeds from distant sources as Google Blog Search, etc.)
They're not acceptable and usually it's very easy to not use them. What's your use case?
I tend to think it is acceptable, as the only case one would try to run the tests offline, he would have to do it explicitly (with mvn -o), and thus would understand that a test method with a name like "testAddRSSFeedFromGoogleBlogSearch" could indeed fail.
The tests should all run in a self contained environment with *minimal* dependencies on the environment. False positives are the plague of tests as we're currently experiencing. 61 continuum mails this morning. We really need to all have a look and fix that as otherwise real problems are hidden and we're loosing faith in our tests. Since today is a bug fixing day we should completely fix our Continuum tests today at the very minimum. Thanks -Vincent
On Jul 23, 2008, at 9:18 AM, Jerome Velociter wrote:
Hello devs,
This is a small wondering I am having while reviewing selenium tests for XWiki Watch. Quoting my last remark on XWATCH-182 :
I am still not sure whether or not tests that needs Internet connectivity are acceptable (such as the tests that add feeds from distant sources as Google Blog Search, etc.)
They're not acceptable and usually it's very easy to not use them. What's your use case?
This test method is not very mandatory : it adds a feed from the Google blog search engine. The "feed adding" feature is already tested via direct (local) URL, which is sufficient IMO.
I tend to think it is acceptable, as the only case one would try to run the tests offline, he would have to do it explicitly (with mvn -o), and thus would understand that a test method with a name like "testAddRSSFeedFromGoogleBlogSearch" could indeed fail.
The tests should all run in a self contained environment with *minimal* dependencies on the environment.
False positives are the plague of tests as we're currently experiencing. 61 continuum mails this morning. We really need to all have a look and fix that as otherwise real problems are hidden and we're loosing faith in our tests.
Since today is a bug fixing day we should completely fix our Continuum tests today at the very minimum.
Good, I'm dedicated to help doing this :) Jérôme.
Thanks -Vincent
_______________________________________________ devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
On Jul 23, 2008, at 10:31 AM, Jerome Velociter wrote:
On Jul 23, 2008, at 9:18 AM, Jerome Velociter wrote:
Hello devs,
This is a small wondering I am having while reviewing selenium tests for XWiki Watch. Quoting my last remark on XWATCH-182 :
I am still not sure whether or not tests that needs Internet connectivity are acceptable (such as the tests that add feeds from distant sources as Google Blog Search, etc.)
They're not acceptable and usually it's very easy to not use them. What's your use case?
This test method is not very mandatory : it adds a feed from the Google blog search engine. The "feed adding" feature is already tested via direct (local) URL, which is sufficient IMO.
Ok this is an interesting use case and is true and valid for all cloud services. The system then becomes the internet. I think we could have these types of tests but we need to make them a special test type and not mix them with the others which are running in a controlled environment since they might be subject to more outages. Thanks -Vincent
I tend to think it is acceptable, as the only case one would try to run the tests offline, he would have to do it explicitly (with mvn -o), and thus would understand that a test method with a name like "testAddRSSFeedFromGoogleBlogSearch" could indeed fail.
The tests should all run in a self contained environment with *minimal* dependencies on the environment.
False positives are the plague of tests as we're currently experiencing. 61 continuum mails this morning. We really need to all have a look and fix that as otherwise real problems are hidden and we're loosing faith in our tests.
Since today is a bug fixing day we should completely fix our Continuum tests today at the very minimum.
Good, I'm dedicated to help doing this :)
Jérôme.
participants (2)
-
Jerome Velociter -
Vincent Massol