[xwiki-devs] Issue with installed count on e.x.o overflowing the AS + too many versions
Hi Thomas and all, Back from holidays! :) I’ve noticed that the new feature of counting installed extensions on e.x.o is having a drawback: it saturates the activity stream, making it very hard to see real edits by users. Every day the scheduler modifies lots of wiki pages to set the new install count. See for example: http://www.xwiki.org/xwiki/bin/view/Main/News I think a simple change would be for the scheduler to make modifications as minor edits. This should prevent the edits from being visible in the AS. WDYT? Now this is going to cause another real issue very soon: pages will soon start to have a lot of revisions and we know this is currently a performance issue. It’s also hiding real edits in the history making the history a bit less clean. I guess an option would be for the scheduler to delete the last revision after it updates a page. Although not very nice, it could work for now. WDYT? Thanks -Vincent
Why does that value have to be stored in the wiki document? Personally, I'd make it a Computed Field, and always get the current value from an external service. This has the disadvantage that the livetable can't filter/order by the number of installations. If the cost of fetching data from an external service is a concern, then add a TTL cache. On 08/15/2016 06:54 AM, Vincent Massol wrote:
Hi Thomas and all,
Back from holidays! :)
I’ve noticed that the new feature of counting installed extensions on e.x.o is having a drawback: it saturates the activity stream, making it very hard to see real edits by users. Every day the scheduler modifies lots of wiki pages to set the new install count. See for example: http://www.xwiki.org/xwiki/bin/view/Main/News
I think a simple change would be for the scheduler to make modifications as minor edits. This should prevent the edits from being visible in the AS.
WDYT?
Now this is going to cause another real issue very soon: pages will soon start to have a lot of revisions and we know this is currently a performance issue. It’s also hiding real edits in the history making the history a bit less clean.
I guess an option would be for the scheduler to delete the last revision after it updates a page. Although not very nice, it could work for now. WDYT?
Thanks -Vincent
-- Sergiu Dumitriu http://purl.org/net/sergiu
Hi Sergiu,
On 15 Aug 2016, at 18:18, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Why does that value have to be stored in the wiki document?
Sure, we could use another store. However the goal of XWiki is also to provide a generic store which can be used to store metadata (through xobjects) related to wiki pages. And install count for an extension really looks like a good metadata candidate for a wiki page. Using another store would also be more work. So, I’d prefer if we could find a way to support this use case which seems legitimate. We have several such use cases, which consist in having some system user perform modifications to wiki pages: * scheduler which updates scheduled jobs with the last fire time xproperty * this extension scheduler which updates install counts * (I’m sure we have more examples but I can’t find one right now ;)) My personal POV is that long term we should have: * Introduce a proper reserved (you cannot create a user named after it) and virtual “System” user (and don’t use “Admin” or “superadmin” which are users which should be used by admins) * Filter out System user changes by default in Activity Stream, History, etc (with options to display them) * Refactor the Versioning store (and possibly get get rid of JRCS) to have a scalable system which supports an unlimited number of revisions without impacting performances * Add an API to merge several revisions into one (this could be useful for auto save too). This is related to what you called “pseudoversion” too: http://extensions.xwiki.org/xwiki/bin/view/Extension/XWiki+Publication+Workf...
Personally, I'd make it a Computed Field, and always get the current value from an external service. This has the disadvantage that the livetable can't filter/order by the number of installations.
If the cost of fetching data from an external service is a concern, then add a TTL cache.
Thanks -Vincent
On 08/15/2016 06:54 AM, Vincent Massol wrote:
Hi Thomas and all,
Back from holidays! :)
I’ve noticed that the new feature of counting installed extensions on e.x.o is having a drawback: it saturates the activity stream, making it very hard to see real edits by users. Every day the scheduler modifies lots of wiki pages to set the new install count. See for example: http://www.xwiki.org/xwiki/bin/view/Main/News
I think a simple change would be for the scheduler to make modifications as minor edits. This should prevent the edits from being visible in the AS.
WDYT?
Now this is going to cause another real issue very soon: pages will soon start to have a lot of revisions and we know this is currently a performance issue. It’s also hiding real edits in the history making the history a bit less clean.
I guess an option would be for the scheduler to delete the last revision after it updates a page. Although not very nice, it could work for now. WDYT?
Thanks -Vincent
-- Sergiu Dumitriu http://purl.org/net/sergiu
On 08/15/2016 12:39 PM, Vincent Massol wrote:
Hi Sergiu,
On 15 Aug 2016, at 18:18, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Why does that value have to be stored in the wiki document?
Sure, we could use another store. However the goal of XWiki is also to provide a generic store which can be used to store metadata (through xobjects) related to wiki pages. And install count for an extension really looks like a good metadata candidate for a wiki page. Using another store would also be more work.
So, I’d prefer if we could find a way to support this use case which seems legitimate. We have several such use cases, which consist in having some system user perform modifications to wiki pages: * scheduler which updates scheduled jobs with the last fire time xproperty * this extension scheduler which updates install counts * (I’m sure we have more examples but I can’t find one right now ;))
My personal POV is that long term we should have: * Introduce a proper reserved (you cannot create a user named after it) and virtual “System” user (and don’t use “Admin” or “superadmin” which are users which should be used by admins) * Filter out System user changes by default in Activity Stream, History, etc (with options to display them) * Refactor the Versioning store (and possibly get get rid of JRCS) to have a scalable system which supports an unlimited number of revisions without impacting performances * Add an API to merge several revisions into one (this could be useful for auto save too). This is related to what you called “pseudoversion” too: http://extensions.xwiki.org/xwiki/bin/view/Extension/XWiki+Publication+Workf...
I agree, but as you said, this is "long term". Perhaps I'm missing something, but I though the count is already stored someplace else. Is the scheduler task that updates these counts not fetching the data from an external place? What I'm proposing, as a quick solution that doesn't require reimplementing the entire storage, user management and activity stream, is that instead of getting the numbers regularly from another service and storing them in the wiki, just let them be fetched directly from there when requested.
Personally, I'd make it a Computed Field, and always get the current value from an external service. This has the disadvantage that the livetable can't filter/order by the number of installations.
If the cost of fetching data from an external service is a concern, then add a TTL cache.
Thanks -Vincent
On 08/15/2016 06:54 AM, Vincent Massol wrote:
Hi Thomas and all,
Back from holidays! :)
I’ve noticed that the new feature of counting installed extensions on e.x.o is having a drawback: it saturates the activity stream, making it very hard to see real edits by users. Every day the scheduler modifies lots of wiki pages to set the new install count. See for example: http://www.xwiki.org/xwiki/bin/view/Main/News
I think a simple change would be for the scheduler to make modifications as minor edits. This should prevent the edits from being visible in the AS.
WDYT?
Now this is going to cause another real issue very soon: pages will soon start to have a lot of revisions and we know this is currently a performance issue. It’s also hiding real edits in the history making the history a bit less clean.
I guess an option would be for the scheduler to delete the last revision after it updates a page. Although not very nice, it could work for now. WDYT?
Thanks -Vincent
-- Sergiu Dumitriu http://purl.org/net/sergiu
devs mailing list devs@xwiki.org http://lists.xwiki.org/mailman/listinfo/devs
-- Sergiu Dumitriu http://purl.org/net/sergiu
On 15 Aug 2016, at 19:11, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
On 08/15/2016 12:39 PM, Vincent Massol wrote:
Hi Sergiu,
On 15 Aug 2016, at 18:18, Sergiu Dumitriu <sergiu@xwiki.org> wrote:
Why does that value have to be stored in the wiki document?
Sure, we could use another store. However the goal of XWiki is also to provide a generic store which can be used to store metadata (through xobjects) related to wiki pages. And install count for an extension really looks like a good metadata candidate for a wiki page. Using another store would also be more work.
So, I’d prefer if we could find a way to support this use case which seems legitimate. We have several such use cases, which consist in having some system user perform modifications to wiki pages: * scheduler which updates scheduled jobs with the last fire time xproperty * this extension scheduler which updates install counts * (I’m sure we have more examples but I can’t find one right now ;))
My personal POV is that long term we should have: * Introduce a proper reserved (you cannot create a user named after it) and virtual “System” user (and don’t use “Admin” or “superadmin” which are users which should be used by admins) * Filter out System user changes by default in Activity Stream, History, etc (with options to display them) * Refactor the Versioning store (and possibly get get rid of JRCS) to have a scalable system which supports an unlimited number of revisions without impacting performances * Add an API to merge several revisions into one (this could be useful for auto save too). This is related to what you called “pseudoversion” too: http://extensions.xwiki.org/xwiki/bin/view/Extension/XWiki+Publication+Workf...
I agree, but as you said, this is "long term".
Perhaps I'm missing something, but I though the count is already stored someplace else. Is the scheduler task that updates these counts not fetching the data from an external place? What I'm proposing, as a quick solution that doesn't require reimplementing the entire storage, user management and activity stream, is that instead of getting the numbers regularly from another service and storing them in the wiki, just let them be fetched directly from there when requested.
I’ve checked the scheduler code and indeed it’s taking the data from ES (see http://extensions.xwiki.org/xwiki/bin/edit/ExtensionCode/UpdateInstalledExte...). So I guess Thomas is saving the data in XWiki for performance reasons and so that we can filter/sort the LT. So indeed with some good caching what you propose could work.However, as you mentioned, we would miss the ability to filter/sort the LT. FTR and FTM I’ve modified the code to save as a minor edit. We now need to solve the version scalability problem. Let’s see what others think. Thanks -Vincent
Personally, I'd make it a Computed Field, and always get the current
value from an external service. This has the disadvantage that the livetable can't filter/order by the number of installations.
If the cost of fetching data from an external service is a concern, then add a TTL cache.
Thanks -Vincent
On 08/15/2016 06:54 AM, Vincent Massol wrote:
Hi Thomas and all,
Back from holidays! :)
I’ve noticed that the new feature of counting installed extensions on e.x.o is having a drawback: it saturates the activity stream, making it very hard to see real edits by users. Every day the scheduler modifies lots of wiki pages to set the new install count. See for example: http://www.xwiki.org/xwiki/bin/view/Main/News
I think a simple change would be for the scheduler to make modifications as minor edits. This should prevent the edits from being visible in the AS.
WDYT?
Now this is going to cause another real issue very soon: pages will soon start to have a lot of revisions and we know this is currently a performance issue. It’s also hiding real edits in the history making the history a bit less clean.
I guess an option would be for the scheduler to delete the last revision after it updates a page. Although not very nice, it could work for now. WDYT?
Thanks -Vincent
-- Sergiu Dumitriu
participants (2)
-
Sergiu Dumitriu -
Vincent Massol