Custom Parsers in IBM QRadar & Juniper STRM

I had some trouble getting all this around my head first time around so I thought I'd put it all together in one place; make things a little bit easier for both myself and whoever else cares to take a look!

Create LSX

First thing is to create a custom parser, also known as a Log Source Extension. This will allow the QRadar to parse custom logs and map data from them to columns in an event.

Do this by looking at all possible logs that will be incoming and specify the data you want to extract, this can include IP addresses, ports, and events. The most important being the event, this is the main component that QRadar will use to identify the log and map it to a QID. 

If you're going to use your own QIDs (which I would recommend) then you're going to want your Event to be mapped to something unique. Meaning if I have the following log:

2015-05-20 00:02:07,518 ERROR [common.ESBLogger] A1217ProducerEmailHandler -- processA1217Resquest -- Mapping not found in lookup table for requirement type code = 1036900347

The actual Event is the fact that the mapping was not found in the lookup table - this is what I'm going to want to look for in my parser. Now when I do something as specific as this, it really limits the amount of regex I actually have to use. If I wanted to use "ERROR" as my event, but also pick up "INFO" and "WARN", then I could use something like: 

(?<=\d\s)([A-Za-z]*)(?=\s\[)

This would result in my events being parsed as either INFO, ERROR, or WARN.

If I knew multiple types of events were going to be produced with the same syntax (following a double hyphen and ending in an equal sign, where Mapping not found in lookup table for requirement type code is in the log) I could use regex to match that:

(?<=\-\s)([A-Za-z\s]*)(?=\s\=)

This would be a bit more useful that just the category of the log.

Example of an LSX

<?xml!version="1.0" encoding="UTFd8"?>
<!--
Device Type: ESBLogger
Protocol: Syslog
-->

<devicedextension!xmlns="event_parsing/device_extension">
<!-- Do not remove the "allEventNames" value -->
<pattern id="allEventNames" xmlns="">
<![CDATA[(.*)]]>
</pattern>
<!-- Everything below this line can be modified --> 

<!-- Match event of Mapping not found -->
<pattern id="EventName1" xmlns="">
<![CDATA[(?<=\-\s)([A-Za-z\s]*)(?=\s\=)]>
</pattern>

<match-group order="1" description="ESBLogger" xmlns="">
<matcher field="EventName" order="1" patterndid="EventName1" capturedgroup="1" enabledsubstitutions="false"/>
<eventdmatchdmultiple patterndid="allEventNames" capturedgroupdindex="1" devicedeventdcategory="unknown" senddidentity="OverrideAndAlwaysSend" />
</match-group>
</devicedextension>

Complete walkthrough:

https://www.sans.org/reading-room/whitepapers/logging/qradar-log-source-extension-walkthrough-35452

Create QID

Once you create a custom parser, you need to map the parsed log to an Event. There are built in events which you can apply, however for complete customization you can create your own. Unfortunately the only way to do this is through the CLI of the QRadar box.

You're going to want to locate the low-level category for the QID map entry you want to create; use the following command for this:

/opt/qradar/bin/qidmap_cli.sh -l

If you want to search for a particular low-level category, you can use the grep command to refine the results; since my example is an application log I'm going to start there

/opt/qradar/bin/qidmap_cli.sh -l | grep Application

Once you know the category you can use the following command to create your QID:

qidmap_cli.sh -c --qname --qdescription --severity --lowlevelcategoryid

The following table provides the utility options; use -h for more options:

Options	Description
-c	Creates a new QID map entry.
--qname	Type the name you want to associate with this QID map entry. The name can be up to 255 characters in length, with no spaces.
--qdescription	Type a description for this QID map entry. The description can be up to 2048 characters in length with no spaces.
--severity	Type the severity level you want to assign to this QID map entry. The valid range is 0 to 10.
--lowlevelcategoryid	Type the low-level category ID you want to assign to this QID map entry.

Complete walkthrough:

https://www.juniper.net/techpubs/software/management/strm/2012_1_R1/strm-managing-qid-map.pdf

Setup Log Source

The first step is to upload the log source extension. There is a log source extension management GUI available within the Admin tab of Qradar. If the log source extension has the proper syntax, it will be accepted when uploaded to the console following the steps under the “add” action.

The next step is to create a generic log source using the “Universal DSM”. The log source needs to match against the data received from the log source. 

If the regular expression used to match against event names is working then the events should start appearing in the Qradar log window. At this point, the event name will show the “unknown” on the Qradar log viewer. Correctly matched fields such as source IP and Destination IP indicate proper functioning of the LSX.

Map Events

Unlike a DSM, a log source extension does not have any built in logic to help with event mapping. Each event shows “unknown”, however, we identified a place within each log to search for some text that will give us information on how to map a particular event. As each unknown event is matched, it will start to show that new event name each time that type of event is seen.

A button titled “Map Event” is available from the menu while viewing an event detail. This will give a pop up box showing the event name we are searching for in the log. There is a search dialog for selecting an available event name or “Qradar Identifier” or QID.

This data will also begin to populate any rules that reference this particular Qradar identifier.