[xwiki-devs] monitoring the cache: how to get the eviction rate?
Hello developers, I have been doing a nifty little cache monitor on www.curriki.org which is still running xwiki 1.5.4. It can show the entries and capacity of the two xwiki caches, pageExistCache and the document cache which sound pretty central. Not surprisingly, they're both full on our production server. name entries capacity pageExistCache 10000 10000 cache 3000 3000 The caches are using OSCache (through "OSCacheCache"). I've been trying to get a real statistical count but there I kind of failed thus far. I can use the StatisticListenerImpl which gives fairly high numbers:
StatisticListenerImpl: Hit = 9583563 / 1320251374, stale hit = 0 / 538, miss = 8541 / 17175236, flush = 27348, entries (added, removed, updates) = 25457741, 0, 7678101 these numbers are integers, and might actually be too small for the 2billions limit of an int, they are static (so count "all the OSCache instances").
What I would like to get is the "eviction rate" but I do not know how to compute it. That is, I wish to read the number of cache-entries which are thrown away because the cache is full and others come in. Once I can have it, I can start tuning (cache size, less greedy algorithms, ...) and measure the effect of such tuning. We have plenty of RAM space to accommodate the cache but we should use this well. The big annoyance with this process is that it can only run on our production server. So thus far I did not dare register a CacheListener in groovy fearing it would suddenly be slow. Has anyone used a different strategy or tool? I'm happy to post my little monitor script. thanks in advance. paul
Hi Paul, On Dec 7, 2011, at 9:05 PM, Paul Libbrecht wrote:
Hello developers,
I have been doing a nifty little cache monitor on www.curriki.org which is still running xwiki 1.5.4.
It can show the entries and capacity of the two xwiki caches, pageExistCache and the document cache which sound pretty central.
Not surprisingly, they're both full on our production server. name entries capacity pageExistCache 10000 10000 cache 3000 3000 The caches are using OSCache (through "OSCacheCache").
I've been trying to get a real statistical count but there I kind of failed thus far.
I can use the StatisticListenerImpl which gives fairly high numbers:
StatisticListenerImpl: Hit = 9583563 / 1320251374, stale hit = 0 / 538, miss = 8541 / 17175236, flush = 27348, entries (added, removed, updates) = 25457741, 0, 7678101 these numbers are integers, and might actually be too small for the 2billions limit of an int, they are static (so count "all the OSCache instances").
What I would like to get is the "eviction rate" but I do not know how to compute it. That is, I wish to read the number of cache-entries which are thrown away because the cache is full and others come in.
Once I can have it, I can start tuning (cache size, less greedy algorithms, ...) and measure the effect of such tuning. We have plenty of RAM space to accommodate the cache but we should use this well.
The big annoyance with this process is that it can only run on our production server. So thus far I did not dare register a CacheListener in groovy fearing it would suddenly be slow.
Has anyone used a different strategy or tool? I'm happy to post my little monitor script.
FYI, newer versions of XE uses JBoss as the default cache implementation. And JBoss offers JMX support. Thus you can get all sort of cache stats using JMX. See here for example: http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Monitoring#HXWikiCachesM... Not sure it helps you but thought you should know (if you're not already aware of it!) Thanks -Vincent
thanks in advance.
paul
Le 7 déc. 2011 à 21:40, Vincent Massol a écrit :
The big annoyance with this process is that it can only run on our production server. So thus far I did not dare register a CacheListener in groovy fearing it would suddenly be slow.
Has anyone used a different strategy or tool? I'm happy to post my little monitor script.
FYI, newer versions of XE uses JBoss as the default cache implementation. And JBoss offers JMX support.
Thus you can get all sort of cache stats using JMX.
Thanks Vincent, I knew a better core meant a better xwiki! Now, Ludovic convinced me that it was possible to upgrade the core and it starts to work: running the "old" gui of curriki.org (called curriki8) on xwiki core 3.2. This is quite exciting. I've explored: http://extensions.xwiki.org/xwiki/bin/view/Extension/JMX+Access which tastes right like I need it. But I am not sure to bridge the gap to obtain the information for the cache. I do not see how to port to the world above what I see at: http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Monitoring#HXWikiCachesM... thanks in advance for hints. Paul
Hello list, With XWiki 3.1 I had managed following the below instructions to use:
MBeanServer server = ManagementFactory.getPlatformMBeanServer(); def mbeanEv = new GroovyMBean(server, 'jboss.cache:service=JBossCache,uniqueId=32dcce09,jmx-resource=CacheMgmtInterceptor') println("mbeanEv: " + mbeanEv + ", enabled: " + mbeanEv.StatisticsEnabled); println("evictions amount: " + mbeanEv.Evictions); println("hits amount: " + mbeanEv.Hits);
where the uniqueId was obtained by listing before. But now that we use XWiki 3.5, I need to do it with Infinispan and it seems a little bit more complex... something close to my lack of competency of mixing Groovy and JMX: I can do this: for(def mbean in server.queryMBeans(new ObjectName("org.infinispan:*"),null)) { out.println("<hr>"); out.println("name : " + mbean.getObjectName()); GroovyMBean mb = new GroovyMBean(server, mbean.getObjectName()); out.println("mb: " + mb); } which does show me the things I can see in JConsole, including an evictionRate for some of the caches. But I do not manage to" - filter the objects that have a statistics - extract this statistics. Hints welcome. paul
The big annoyance with this process is that it can only run on our production server. So thus far I did not dare register a CacheListener in groovy fearing it would suddenly be slow.
Has anyone used a different strategy or tool? I'm happy to post my little monitor script.
FYI, newer versions of XE uses JBoss as the default cache implementation. And JBoss offers JMX support.
Thus you can get all sort of cache stats using JMX.
Thanks Vincent,
I knew a better core meant a better xwiki!
Now, Ludovic convinced me that it was possible to upgrade the core and it starts to work: running the "old" gui of curriki.org (called curriki8) on xwiki core 3.2. This is quite exciting.
I've explored: http://extensions.xwiki.org/xwiki/bin/view/Extension/JMX+Access which tastes right like I need it.
But I am not sure to bridge the gap to obtain the information for the cache.
I do not see how to port to the world above what I see at: http://platform.xwiki.org/xwiki/bin/view/AdminGuide/Monitoring#HXWikiCachesM...
Le 7 déc. 2011 à 21:05, Paul Libbrecht a écrit :
I can use the StatisticListenerImpl which gives fairly high numbers:
StatisticListenerImpl: Hit = 9583563 / 1320251374, stale hit = 0 / 538, miss = 8541 / 17175236, flush = 27348, entries (added, removed, updates) = 25457741, 0, 7678101 these numbers are integers, and might actually be too small for the 2billions limit of an int, they are static (so count "all the OSCache instances").
And it did spin to negative already several times. How delicious!
What I would like to get is the "eviction rate" but I do not know how to compute it. That is, I wish to read the number of cache-entries which are thrown away because the cache is full and others come in.
I simply counted differences between snapshots the number of adds minus the capacity. If the caches are full (which is the case after about 2 minutes after starting curriki), this gives a faithful estimate. Until the number of adds becomes negative ;->.
Once I can have it, I can start tuning (cache size, less greedy algorithms, ...) and measure the effect of such tuning. We have plenty of RAM space to accommodate the cache but we should use this well.
Well, we just had a restart because the appserv was sweating 1700% and had 23 threads executing the same place: at com.opensymphony.oscache.base.Cache.dispatchCacheMapAccessEvent(Cache.java:907) (somewhere behind a getFromCache) Trying to understand what would have caused this, I could not understand it.
The big annoyance with this process is that it can only run on our production server. So thus far I did not dare register a CacheListener in groovy fearing it would suddenly be slow.
Can anyone state me a danger of doing so?
Has anyone used a different strategy or tool?
I've discovered that Zabbix would have a time-synched view (or close to it). Did anyone try Zabbix with XWiki? Indeed Vincent, that would leverage JMX so it could be used directly for XWiki. Can the eviction-rate be computed with this? I was surprised to find the tool-set of OSCache's statistics quite meager.
I'm happy to post my little monitor script.
The cache monitor is here: http://www.curriki.org/xwiki/bin/view/CurrikiCode/CacheMonitor (publicly accessible) with the stats disabled. thanks in advance Paul
participants (2)
-
Paul Libbrecht -
Vincent Massol