As you can see from the stack trace below, properties seem to be the issue in this case. The Java 1.4 SAXParser JavaDoc clearly states that "An implementation of SAXParser is NOT guaranteed to behave as per the specification if it is used concurrently by two or more threads." More recent versions of the JavaDoc have removed the warning, though the presence of "setProperty()" certainly means that a SAXParser is not immutable. As a result, a single SAXParser instance can end up simultaneously parsing documents in multiple threads.
A quick look at the source code shows that an AutoDetectParser holds a MimeTypes which holds an XmlRootExtractor which holds a SAXParser. Other posts have stated that AutoDetectParser is thread-safe. This seems to be a concurrency problem we have not seen the issue when running single threaded. In our case, it appears that a HashMap inside Xerces gets corrupted, causing an infinite loop inside HashMap.get(). If we assign multiple threads to the parsing task we find that the AutoDetectParser.parse() method occasionally fails to return. We are using Tika 0.5 to parse files that are added to a Lucene index.