[xwiki-users] Solr Memory Issues
Hi, I'm experiencing frequent crashes that I think are related to Solr, and I'd appreciate any advice on how to fix them. I have a fairly large wiki (with several thousand attachments) running 5.4.3 on tomcat7, and it goes down multiple times a day. Sometimes it will remain unresponsive for a few minutes then come back on its own, other times it stays down until I restart tomcat. Usage runs anywhere from a few active users to several thousand, depending on time of day. Crashes happen even when the number of users is low. Checking catalina.out and the localhost_access logs, there almost always appears to be requests related to solr around the time it goes down. Sometimes catalina.out shows a java heap out of memory error, but other times there are no errors in the logs at all, the site is just unresponsive. I can usually reproduce the issue by running several searches back to back, while browsing pages in another tab. While the searches are in progress, the responsiveness of browsing pages steadily drops until it times out. It usually only takes 5 or 10 searches before this happens. The other thing that makes me suspect Solr is that even when the site doesn't crash, while running a search in one tab, requests in the other tab don't finish until the search is complete - it (subjectively, I know) feels like other requests are suspended while Solr is thinking. Potentially related is that I occasionally also see JDBC connection pool related errors in the logs when it goes down - I have tried tweaking my connection pool size settings, to no avail. My current java memory settings are: JAVA_OPTS="-Djava.awt.headless=true -Xms2500m -Xmx2500m -XX:PermSize=64m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC" I've used everything from the 800m xwiki.org settings to 3GB+, and it doesn't seem to make a difference. I have also reduced the cache settings from 3000 back to 100 in xwiki.properties, which hasn't helped. My hibernate settings are: connection.pool.size = 100 statement_cache.size = 50 dbcp.maxActive = 100 dbcp.maxIdle = 10 dbcp.maxWait = 300000 Mysql max_connections is set to 290 I have changed nearly all of those numbers at some point, but nothing seems to help. Any assistance is greatly appreciated - even just tips on how to troubleshoot what's actually taking the site down, since the logs aren't always useful. thanks, aaron
It's not exactly in SOLR but it's triggered by SOLR when it's rendering the pages titiles. See http://jira.xwiki.org/browse/XWIKI-10382. The fix is part of the comming 5.4.6. On Fri, Sep 5, 2014 at 7:13 PM, Aaron Hardy at AC <ashtarcommunications@gmail.com> wrote:
Hi,
I'm experiencing frequent crashes that I think are related to Solr, and I'd appreciate any advice on how to fix them.
I have a fairly large wiki (with several thousand attachments) running 5.4.3 on tomcat7, and it goes down multiple times a day. Sometimes it will remain unresponsive for a few minutes then come back on its own, other times it stays down until I restart tomcat. Usage runs anywhere from a few active users to several thousand, depending on time of day. Crashes happen even when the number of users is low.
Checking catalina.out and the localhost_access logs, there almost always appears to be requests related to solr around the time it goes down. Sometimes catalina.out shows a java heap out of memory error, but other times there are no errors in the logs at all, the site is just unresponsive.
I can usually reproduce the issue by running several searches back to back, while browsing pages in another tab. While the searches are in progress, the responsiveness of browsing pages steadily drops until it times out. It usually only takes 5 or 10 searches before this happens.
The other thing that makes me suspect Solr is that even when the site doesn't crash, while running a search in one tab, requests in the other tab don't finish until the search is complete - it (subjectively, I know) feels like other requests are suspended while Solr is thinking.
Potentially related is that I occasionally also see JDBC connection pool related errors in the logs when it goes down - I have tried tweaking my connection pool size settings, to no avail.
My current java memory settings are: JAVA_OPTS="-Djava.awt.headless=true -Xms2500m -Xmx2500m -XX:PermSize=64m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC"
I've used everything from the 800m xwiki.org settings to 3GB+, and it doesn't seem to make a difference.
I have also reduced the cache settings from 3000 back to 100 in xwiki.properties, which hasn't helped.
My hibernate settings are: connection.pool.size = 100 statement_cache.size = 50 dbcp.maxActive = 100 dbcp.maxIdle = 10 dbcp.maxWait = 300000
Mysql max_connections is set to 290
I have changed nearly all of those numbers at some point, but nothing seems to help.
Any assistance is greatly appreciated - even just tips on how to troubleshoot what's actually taking the site down, since the logs aren't always useful.
thanks,
aaron _______________________________________________ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
-- Thomas Mortagne
Thanks for the info. I am planning to move the farm to 6.2 anyway, so hopefully the issue will be fixed after the upgrade. I'll ask again if I have trouble keeping it stable on the most recent version. On Sat, Sep 6, 2014 at 1:00 AM, Thomas Mortagne <thomas.mortagne@xwiki.com> wrote:
It's not exactly in SOLR but it's triggered by SOLR when it's rendering the pages titiles. See http://jira.xwiki.org/browse/XWIKI-10382. The fix is part of the comming 5.4.6.
On Fri, Sep 5, 2014 at 7:13 PM, Aaron Hardy at AC <ashtarcommunications@gmail.com> wrote:
Hi,
I'm experiencing frequent crashes that I think are related to Solr, and I'd appreciate any advice on how to fix them.
I have a fairly large wiki (with several thousand attachments) running 5.4.3 on tomcat7, and it goes down multiple times a day. Sometimes it will remain unresponsive for a few minutes then come back on its own, other times it stays down until I restart tomcat. Usage runs anywhere from a few active users to several thousand, depending on time of day. Crashes happen even when the number of users is low.
Checking catalina.out and the localhost_access logs, there almost always appears to be requests related to solr around the time it goes down. Sometimes catalina.out shows a java heap out of memory error, but other times there are no errors in the logs at all, the site is just unresponsive.
I can usually reproduce the issue by running several searches back to back, while browsing pages in another tab. While the searches are in progress, the responsiveness of browsing pages steadily drops until it times out. It usually only takes 5 or 10 searches before this happens.
The other thing that makes me suspect Solr is that even when the site doesn't crash, while running a search in one tab, requests in the other tab don't finish until the search is complete - it (subjectively, I know) feels like other requests are suspended while Solr is thinking.
Potentially related is that I occasionally also see JDBC connection pool related errors in the logs when it goes down - I have tried tweaking my connection pool size settings, to no avail.
My current java memory settings are: JAVA_OPTS="-Djava.awt.headless=true -Xms2500m -Xmx2500m -XX:PermSize=64m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC"
I've used everything from the 800m xwiki.org settings to 3GB+, and it doesn't seem to make a difference.
I have also reduced the cache settings from 3000 back to 100 in xwiki.properties, which hasn't helped.
My hibernate settings are: connection.pool.size = 100 statement_cache.size = 50 dbcp.maxActive = 100 dbcp.maxIdle = 10 dbcp.maxWait = 300000
Mysql max_connections is set to 290
I have changed nearly all of those numbers at some point, but nothing seems to help.
Any assistance is greatly appreciated - even just tips on how to troubleshoot what's actually taking the site down, since the logs aren't always useful.
thanks,
aaron _______________________________________________ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
-- Thomas Mortagne _______________________________________________ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
On Fri, Sep 5, 2014 at 8:13 PM, Aaron Hardy at AC <ashtarcommunications@gmail.com> wrote:
Hi,
I'm experiencing frequent crashes that I think are related to Solr, and I'd appreciate any advice on how to fix them.
I have a fairly large wiki (with several thousand attachments) running 5.4.3 on tomcat7, and it goes down multiple times a day. Sometimes it will remain unresponsive for a few minutes then come back on its own, other times it stays down until I restart tomcat. Usage runs anywhere from a few active users to several thousand, depending on time of day. Crashes happen even when the number of users is low.
Checking catalina.out and the localhost_access logs, there almost always appears to be requests related to solr around the time it goes down. Sometimes catalina.out shows a java heap out of memory error, but other times there are no errors in the logs at all, the site is just unresponsive.
I can usually reproduce the issue by running several searches back to back, while browsing pages in another tab. While the searches are in progress, the responsiveness of browsing pages steadily drops until it times out. It usually only takes 5 or 10 searches before this happens.
Are you referring to the live search (a.k.a. search suggest) from the top right corner of each page or are you using the dedicated search page, Main.Search? One big difference is that the live search performs multiple queries (one per each search suggest source configured in the administration) while the dedicated search page performs a single query. Also, the default search suggest sources have been improved in 5.4.2 as part of http://jira.xwiki.org/browse/XWIKI-10051 . Maybe you kept the old search suggest sources when you upgraded to 5.4.3? Hope this helps, Marius
The other thing that makes me suspect Solr is that even when the site doesn't crash, while running a search in one tab, requests in the other tab don't finish until the search is complete - it (subjectively, I know) feels like other requests are suspended while Solr is thinking.
Potentially related is that I occasionally also see JDBC connection pool related errors in the logs when it goes down - I have tried tweaking my connection pool size settings, to no avail.
My current java memory settings are: JAVA_OPTS="-Djava.awt.headless=true -Xms2500m -Xmx2500m -XX:PermSize=64m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC"
I've used everything from the 800m xwiki.org settings to 3GB+, and it doesn't seem to make a difference.
I have also reduced the cache settings from 3000 back to 100 in xwiki.properties, which hasn't helped.
My hibernate settings are: connection.pool.size = 100 statement_cache.size = 50 dbcp.maxActive = 100 dbcp.maxIdle = 10 dbcp.maxWait = 300000
Mysql max_connections is set to 290
I have changed nearly all of those numbers at some point, but nothing seems to help.
Any assistance is greatly appreciated - even just tips on how to troubleshoot what's actually taking the site down, since the logs aren't always useful.
thanks,
aaron _______________________________________________ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
I've seen queries to both suggest and dedicated in the logs around the times when it goes down - But I assume most users start by using the search suggest, and then move to the dedicated page if they don't find what they need, so search suggest is probably triggered in most circumstances. For what it's worth, my short term fix has been to hide the search box (which would force users to go to Main.Search if they want to search for something), and it's been much more stable. I haven't seen any search-related issues so far on 6.2 RC1, so I'm hopeful that upgrading this farm will deal with the issue as well. I'm pretty sure I just used the deb package for 5.4.3, so it's unlikely I have old search suggest sources, but who knows. Thanks for the info, aaron On Mon, Sep 15, 2014 at 8:38 AM, Marius Dumitru Florea < mariusdumitru.florea@xwiki.com> wrote:
On Fri, Sep 5, 2014 at 8:13 PM, Aaron Hardy at AC <ashtarcommunications@gmail.com> wrote:
Hi,
I'm experiencing frequent crashes that I think are related to Solr, and I'd appreciate any advice on how to fix them.
I have a fairly large wiki (with several thousand attachments) running 5.4.3 on tomcat7, and it goes down multiple times a day. Sometimes it will remain unresponsive for a few minutes then come back on its own, other times it stays down until I restart tomcat. Usage runs anywhere from a few active users to several thousand, depending on time of day. Crashes happen even when the number of users is low.
Checking catalina.out and the localhost_access logs, there almost always appears to be requests related to solr around the time it goes down. Sometimes catalina.out shows a java heap out of memory error, but other times there are no errors in the logs at all, the site is just unresponsive.
I can usually reproduce the issue by running several searches back to back, while browsing pages in another tab. While the searches are in progress, the responsiveness of browsing pages steadily drops until it times out. It usually only takes 5 or 10 searches before this happens.
Are you referring to the live search (a.k.a. search suggest) from the top right corner of each page or are you using the dedicated search page, Main.Search? One big difference is that the live search performs multiple queries (one per each search suggest source configured in the administration) while the dedicated search page performs a single query. Also, the default search suggest sources have been improved in 5.4.2 as part of http://jira.xwiki.org/browse/XWIKI-10051 . Maybe you kept the old search suggest sources when you upgraded to 5.4.3?
Hope this helps, Marius
The other thing that makes me suspect Solr is that even when the site doesn't crash, while running a search in one tab, requests in the other
tab
don't finish until the search is complete - it (subjectively, I know) feels like other requests are suspended while Solr is thinking.
Potentially related is that I occasionally also see JDBC connection pool related errors in the logs when it goes down - I have tried tweaking my connection pool size settings, to no avail.
My current java memory settings are: JAVA_OPTS="-Djava.awt.headless=true -Xms2500m -Xmx2500m -XX:PermSize=64m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC"
I've used everything from the 800m xwiki.org settings to 3GB+, and it doesn't seem to make a difference.
I have also reduced the cache settings from 3000 back to 100 in xwiki.properties, which hasn't helped.
My hibernate settings are: connection.pool.size = 100 statement_cache.size = 50 dbcp.maxActive = 100 dbcp.maxIdle = 10 dbcp.maxWait = 300000
Mysql max_connections is set to 290
I have changed nearly all of those numbers at some point, but nothing seems to help.
Any assistance is greatly appreciated - even just tips on how to troubleshoot what's actually taking the site down, since the logs aren't always useful.
thanks,
aaron _______________________________________________ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
participants (3)
-
Aaron Hardy at AC -
Marius Dumitru Florea -
Thomas Mortagne