Issue Details (XML | Word | Printable)

Key: SCAR-7
Type: Bug Bug
Status: Resolved Resolved
Resolution: Fixed
Priority: Major Major
Assignee: Sergio Bossa
Reporter: Quoc-Thuan Ho
Votes: 0
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Scarlet - Clustering for Jira

writelock, threaddump, lock timeout... while submitting JIRA issues

Created: 06/Mar/08 07:58 PM   Updated: 11/Mar/08 05:28 AM
Component/s: Lucene Indexes
Affects Version/s: 1.0RC1
Fix Version/s: 1.0RC2

Time Tracking:
Not Specified

Environment: Jira + Scarlet + Terracotta on Linux with NFS
Issue Links:
Cause
 

Labels:


 Description  « Hide
These are JIRA's Tomcat exceptions we got while submitting new issues to a JIRA cluster. About 20-40% number of submitted issues have not been created.

(1)

2008-03-06 16:13:10,976 http-8080-Processor23 ERROR [event.listeners.search.IssueIndexListener] Error re-indexing Issue[GenericEntity:Issue][created,2008-03-06 16:13:08.764][summary,New issue][workflowId,11648][timeoriginalestimate,null][type,3][timespent,null][environment,Linux][status,1][updated,2008-03-06 16:13:08.764][id,10588][timeestimate,null][key,PEIGHT-78][description,Create issue using jmeter][duedate,null][reporter,local_jira_admin][project,10038][security,null][votes,0][assignee,null][priority,2]
com.atlassian.jira.issue.index.IndexException: com.atlassian.bonnie.LuceneException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SingleInstanceLock: write.lock
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:292)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:244)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndex(DefaultIndexManager.java:264)
at sun.reflect.GeneratedMethodAccessor306.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:71)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
at $Proxy4.reIndex(Unknown Source)
at com.atlassian.jira.event.listeners.search.IssueIndexListener.reIndex(IssueIndexListener.java:126)
at com.atlassian.jira.event.listeners.search.IssueIndexListener.reIndex(IssueIndexListener.java:136)
at com.atlassian.jira.event.listeners.search.IssueIndexListener.issueCreated(IssueIndexListener.java:51)
at com.atlassian.jira.event.issue.AbstractIssueEventListener.workflowEvent(AbstractIssueEventListener.java:110)
at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:108)
at com.atlassian.jira.event.issue.IssueEventDispatcher.dispatchEvent(IssueEventDispatcher.java:89)
at com.atlassian.jira.workflow.function.event.FireIssueEventFunction.execute(FireIssueEventFunction.java:60)
at com.opensymphony.workflow.AbstractWorkflow.executeFunction(AbstractWorkflow.java:869)
at com.opensymphony.workflow.AbstractWorkflow.transitionWorkflow(AbstractWorkflow.java:1265)
at com.opensymphony.workflow.AbstractWorkflow.initialize(AbstractWorkflow.java:618)
at com.atlassian.jira.workflow.SimpleWorkflowManager.createIssue(SimpleWorkflowManager.java:189)
at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssue(DefaultIssueManager.java:365)
at com.atlassian.jira.issue.managers.DefaultIssueManager.createIssue(DefaultIssueManager.java:313)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:71)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
at $Proxy3.createIssue(Unknown Source)
at com.atlassian.jira.web.action.issue.CreateIssueDetails.createIssue(CreateIssueDetails.java:141)
at com.atlassian.jira.web.action.issue.CreateIssueDetails.doExecute(CreateIssueDetails.java:110)
at webwork.action.ActionSupport.execute(ActionSupport.java:153)
at com.atlassian.jira.action.JiraActionSupport.execute(JiraActionSupport.java:54)
at webwork.dispatcher.GenericDispatcher.executeAction(GenericDispatcher.java:132)
at com.atlassian.jira.web.dispatcher.JiraServletDispatcher.service(JiraServletDispatcher.java:211)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.jira.web.filters.AccessLogFilter.doFilter(AccessLogFilter.java:73)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:119)
at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:55)
at com.atlassian.jira.web.filters.SitemeshExcludePathFilter.doFilter(SitemeshExcludePathFilter.java:38)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:192)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.seraph.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:114)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:110)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.util.profiling.filters.ProfilingFilter.doFilter(ProfilingFilter.java:132)
at com.atlassian.jira.web.filters.JIRAProfilingFilter.doFilter(JIRAProfilingFilter.java:16)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.jira.web.filters.ActionCleanupDelayFilter.doFilter(ActionCleanupDelayFilter.java:43)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.jira.web.filters.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:50)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:72)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:350)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.gzipfilter.GzipFilter.doFilter(GzipFilter.java:89)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.core.filters.AbstractEncodingFilter.doFilter(AbstractEncodingFilter.java:37)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.atlassian.jira.appconsistency.db.DatabaseCompatibilityEnforcerFilter.doFilter(DatabaseCompatibilityEnforcerFilter.java:39)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:541)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:869)
at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:664)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:80)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
....

(2)

2008-03-06 16:19:32,425 http-8080-Processor19 ERROR [jdk.utilities.threaddump.ThreadDumper] sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
2008-03-06 16:19:32,425 http-8080-Processor19 ERROR [jdk.utilities.threaddump.ThreadDumper] sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
2008-03-06 16:19:32,425 http-8080-Processor19 ERROR [jdk.utilities.threaddump.ThreadDumper] java.lang.reflect.Method.invoke(Method.java:585)
2008-03-06 16:19:32,425 http-8080-Processor19 ERROR [jdk.utilities.threaddump.ThreadDumper] org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:294)
2008-03-06 16:19:32,425 http-8080-Processor19 ERROR [jdk.utilities.threaddump.ThreadDumper] org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:432)
.....

(3)

2008-03-06 16:19:32,433 http-8080-Processor19 ERROR [jira.issue.index.DefaultIndexManager] Could not reindex: com.atlassian.jira.issue.util.IssueGVsIssueIterable (1 items): [PNINE-3]
2008-03-06 16:19:32,519 http-8080-Processor25 ERROR [jira.issue.index.DefaultIndexManager] Lock attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Lock timout - waited 30000 milliseconds
at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:451)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:279)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:244)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndex(DefaultIndexManager.java:264)



 All   Comments   Work Log   Change History   FishEye   Crucible   Builds      Sort Order: Ascending order - Click to sort in descending order
[ Permlink | « Hide ]
Sergio Bossa added a comment - 07/Mar/08 03:09 AM
Problems number (2) and (3) are unrelated to Scarlet: they're due to internal lock timeouts in Jira.
Problem number (1) seems to be related to a Scarlet problem while managing com.atlassian.bonnie.ConcurrentLuceneConnection locks.

[ Permlink | « Hide ]
Sergio Bossa added a comment - 07/Mar/08 03:10 AM
John,

What do you put on NFS?


[ Permlink | « Hide ]
Quoc-Thuan Ho added a comment - 07/Mar/08 03:18 AM
Sergio,

I used NFS for jira, terracotta, and scarlet source.
All 8 nodes shared this NFS.


[ Permlink | « Hide ]
Sergio Bossa added a comment - 07/Mar/08 03:40 AM
John,

Why did you do that?
Even if unrelated to this issue, I strongly suggest you to remove any use of NFS and put a separate copy of Jira, Terracotta and Scarlet on every node, locally on its filesystem.


[ Permlink | « Hide ]
Sergio Bossa added a comment - 08/Mar/08 01:49 AM
Just to clarify my previous comment: you need a shared filesystem area, i.e. NFS, only for attachments and backups directories; these are the only directories that you need to share among all Jira nodes.

Cheers,

Sergio B.


[ Permlink | « Hide ]
Sergio Bossa added a comment - 11/Mar/08 05:28 AM
Fixed by SCAR-8.


Powered by Atlassian JIRA the Professional Issue Tracker. (Enterprise Edition, Version: 3.13.2-#335) - Bug/feature request - Atlassian news - Contact Administrators