UBA went AWAL, seem's like it got corrupted.
What would be the possible reason for UBA to go AWAL?
Since three days, the main UBA Dashboard was working fine but when trying to inspect users, nothing was showing up, something like queries where not giving detailed activity. When replaying queries in Advanced log search that where present in the UBA log, those where not working in the activity log.
Anyhow killing the run.py (as suggested by @matthew.ouelette ) did the trick, i was just wondering if other people had the same issues or what could cause such issues.
Which version was this? In 1.2 we fixed some issues around how we handle the asynchronous searching. In 1.1 and before it was possible to hit some deadlock scenarios with how flask was handling the search requests. Did you have app logs from before/during you hit this error? I would be curious to see any exceptions/errors in app.log and poll.log.
1.2. Found add stuff like:
Nov 22 03:22:02 127.0.0.1 [APP_ID/1302][NOT:0000003000][ERROR] Failed to perform search: "select count() as total from events where category between 24000 and 25000 and qidname(qid)='Sense Offense Inject' last 1 hours" Nov 22 03:33:02 127.0.0.1 [APP_ID/1302][NOT:0000003000][ERROR] Failed to perform search: "select count() as total from events where category between 24000 and 25000 and senseValue is not null and not ReferenceSetContains('UBA : Trusted Usernames', username) last 1 hours" app.log:Nov 24 18:17:00 127.0.0.1 [APP_ID/1302][NOT:0000003000][ERROR] Failed to generate dashboard top panel
app.log:Nov 24 18:18:40 127.0.0.1 [APP_ID/1302][NOT:0000003000][ERROR] Failed to perform search: u'select CATEGORYNAME(category) as parent, qidname(qid) as child, sum(senseValu e) as total from events where senseValue is not null and category between 24000 and 25000 and ( username = {{XXXXXXXXXXXXXX}} ) group by parent, child last 1 hours'
bash-4.1# grep -i error poll.log 2016-11-21 03:04:14,317 [com.ibm.InsiderThreat] [ERROR] - Unable to send user score event - sendto() takes exactly 3 arguments (2 given) 2016-11-22 09:21:31,081 [com.ibm.InsiderThreat] [ERROR] - Poll QRadar reference table XXXXX failed; ..