Version
5.2.5
Issue
Searches in Process Search page consistently return 'No Results', recent Event data is also missing from the Console.
Symptoms
Nearly all searches in the Process Search page return very quickly with 'No Results'.
Additionally, the following, or similar messages are written to the /var/log/cb/solr/debug.log file:
2017-06-30 11:59:54,920 - [ERROR] - from org.apache.solr.update.UpdateHandler in http-8080-2
Process doc with id 0000455d-0000-0364-01d2-f1b09a0c8909-00000001 has a mismatch on update: field: cmdline does not match: old value = c:\windows\system32\svchost.exe -k localservicenetworkrestricted new value = C:\Windows\System32\svchost.exe -k LocalServiceNetworkRestricted
2017-06-30 08:35:45,673 - [ERROR] - from org.apache.solr.update.UpdateHandler in http-8080-11 Process doc with id 000035ee-0000-2ee8-01d2-f195a08e6b29-00000001 has a mismatch on update: field: process_name does not match: old value = chrome.exe new value = conhost.exe, field: path does not match: old value = c:\program files\google\chrome\application\chrome.exe new value = c:\windows\system32\conhost.exe, field: process_md5 does not match: old value = a7ed399087929faf32dce43f07a9fe3e new value = 19bd5196020e5d015e224905bbf7c8a1 |
Cause
These messages are caused by a heavy flow of incoming 'CrossProcEvents'. In 5.2.x, we need to re-parse events that are already stored in a process doc for their timestamp, when we split the doc into segments the events will be stored in each segment in chronological order. When we read in the older events, we use the 'Jackson' parser that does conversion to/from CSV and JSON. When 'Jackson' does the parsing, it locks an event-type-specific singleton instance of an ObjectReader object, so when there are many events to parse, the locking slows data ingestion considerably.
Solution
The resolution to this issue is to apply the v6.1 update, as this code uses a pool of ObjectReaders per event type instead of a single one.
