Collector Development Topics

COLLECTOR CUSTOMIZATION

Collector Customization

This site is primarily about developing new Collectors based on the SDK template. In some circumstances, however, customers wish to simply extend the operation of an existing Collector, in which case the Collector will operate in what we call custom mode. This special mode will load your custom logic and apply it to the running Collector, but can easily be disabled as needed.

NOTE: Custom code will be executed and applied to all events that are parsed by the existing Collector. You can extend the Collector to support new events, or you can enhance the Collector to add additional data to existing supported events. Events that are not parseable, however, will not trigger custom code (e.g., events for which parsing is short-circuited by calling sendUnsupported()).

Adding Custom Code to Collectors

The first step in adding extension parsing logic to existing GA (Generally Available) Collectors is to spend a little time researching the Collector you want to modify - each developer will have written to their own tastes, and you will have to determine which variables you want to manipulate. Review the top-level Collector state diagram and notice the yellow states - those represent the code points at which you can inject your custom code. The intent is to allow you to:

Perform additional initialization steps like loading your own maps, setting static event values, and setting up the environment.
Do additional pre-parsing before the core Collector code parses the data, in case local configuration has modified the input expected by the Collector.
Do additional post-parsing on the event data to extract additional information, set other event fields, or inject additional data.

In order for your custom code to work correctly, you will need to understand what the Collector expects to see on its input - e.g. what it expects the input rec object to look like - and what the Collector expects the rec object to look like after parsing. One of the best ways to understand the Collector is to run it in the ESM debugger, and simply to watch what the rec object looks like at different points in the process. In particular, you will want to look at the rec object just after you enter the preParse() state, and just before you exit the normalize() state (you can put breakpoints on those method calls at the bottom of the debugger script file).

Setting Up to Add Custom Code

The basic overview on how to add custom logic to an existing Collector is: Add Auxiliary File

Download the SDK and install it.
Locate the file //current/sdk/common/Collector/src/custom.js and make a copy of it (anywhere, it doesn't matter).
Edit the file to define at least the three supported custom parsing methods (see below).
Create any other artifacts you will need to add to your Collector (maps, etc).
Start up the Sentinel Control Center and find the Collector you want to extend in the Script pane.
Select the "Add Auxiliary File" button and import your custom.js file into the Collector plug-in (DO NOT rename the file!). Also import your other artifacts the same way.
Find any deployed Collector instances, right-click on them, and select "Edit".
On the Parameters tab, find the Execution Mode parameter and change it to custom.
Restart the Collector — your logic should be applied

Creating Custom Collector Methods

Before you begin, you must understand how the custom methods fit into the parsing methodology of the Collector execution environment and how they interact with the other Collector methods. The examples presented here are not guaranteed to work for all GA Collectors, especially if those Collectors use Sessions or other advanced parsing techniques. That said, these examples should work for most common cases.

customInit()

[Extends Collector] This method allows you to set up additional features in the Collector execution environment. You can:

Load additional parameters
Load additional DataMaps and KeyMaps
Initialize external systems
Predefine static event fields
and so forth...

Collector.prototype.customInit = function() {  
    // Define a map from from the IP to system owner (not using Asset tables or the Mapping Service)  
    this.MAPS.ownerMap = new KeyMap(this.CONFIG.collDir + "/sysowners.map");
    // Define a static event field (the protoEvt object is cloned at the beginning of processing for each new Record
    this.protoEvt.customerVar35 = "HR Department Windows Collector";
    // To extend or replace entries in an existing KeyMap (you will need to look at the existing Collector to determine what maps it uses)
    this.MAPS.sevMap.extend(this.CONFIG.collDir +  "/customsev.map");
    // To extend the existing Rec2Evt map (defines how Record attributes are mapped to the output Event)
    this.MAPS.Rec2Evt.extend(this.CONFIG.collDir +  "/customR2E.map");
    return true; 
};

customPreparse(): [Extends Record] This method allows you to apply additional pre-parsing logic for the Collector. You might use this if, for example, you are tunneling standard event data through some other channel, and an extra header is applied. You can strip that off here before passing the standard message to the normal pre-parse method.

Record.prototype.customPreparse = function(e) {  
    // Our syslog relay violates RFC3164 and inserts a custom header  
    this.s_RXBufferString = this.s_RXBufferString.substr(this.s_RXBufferString.indexOf("<"));
    return true;  
};

To do this effectively, you need to know exactly what the input will look like when it arrives from the Connector. The best way to do this is to examine the output from a Connector Dump file in the ESM debugger.

customParse()

[Extends Record] This method allows you to define additional parsing logic to be applied to inbound records. This is applied after the normal parse() and normalize() methods, after the normal Collector parsing has been completed. You can:

Map additional data into the event
Parse out additional data which is significant only in your enterprise environment
Overwrite existing parsing methods
Extend available parsing methods to support new data types and events

Record.prototype.customParse = function(e) {  
    // Look up the owner for each destination IP (you will also need to add CustomerVar65 to Rec2Evt.map, see above) 
    this.cv65 = this.MAPS.ownerMap.lookup(this.dip);
    this.parseCustomField(e);  // see below
    return true;
};

anythingyouwant(): [Typically extends Record] In fact, you can define any additional parsing methods you want within the custom.js file. The general form is:

/**  
 * This method parses custom fields from the event 
 * @param {Event} e  The output event, passed in so you can call methods like e.add2EI() 
 */ 
Record.prototype.parseCustomField = function(e) {
    this.s_RXBufferString.search(/Critcality=(\S*));
    // You also need to add a new mapping for CustomerVar78 in an extension to Rec2Evt.map, see above  
    this.cv78 = RegExp.$1;
    return true;  
};

Examples

Many Collectors use event data to select parsers, for example they might construct a method name from the "appid" of a syslog message, as follows:

Record.prototype.parse() {  
    if ( typeof this.["parse-" + this.appid] != "undefined" ) {   
        this["parse-" + this.appid](e);
    }  
};

The above code would call the following method if rec.appid were set to sshd:

Record.prototype["parse-sshd"] = function(e) {  
    // parse sshd data  
};

For Collectors of this type (which include many of the GA Collectors) you can extend the domain of what apps are parsed simply by adding your own parsers — there is no need to modify the customParse() method at all. The if clause above will find the new parser and pass control to it.

If you wish to customize an existing Collector, you will have to review the way in which that Collector is implemented, and look for convenient extension points similar to the above example.

Back to Develop to Sentinel