Object type: Data sync policy object | ||
---|---|---|
Information about the data sync policy | ||
Field | Required | Description |
alwaysSchedule | false | Boolean flag specifying whether data syncing is to run continuously ("true") or according to a schedule ("false"). If "false", the ScheduleItems field specifies the schedule to be used for syncing data. |
backOffPeriod | false | The number of seconds to backoff between queries when the synced-to time is up to date. |
countColumn | false | Column in destination table where event counts are to be stored. The type of this column should be capable of storing an integer. |
dbConnectionConfig | false | This is a nested JSON <B><Database Connection></B> object that specifies the destination database to sync data to. It is used in conjunction with the table field to specify the destination table events are to be synced to. See below for description of fields in Database Connection objects. NOTE: If the policy is associated with a report, this cannot be changed - whatever is specified here will be ignored. |
doSummaries | false | Boolean flag specifying whether this policy should do event summaries instead of syncing individual events. If omitted, this defaults to false. If true, then the <B>summaryPeriod</B>, <B>countColumn</B>, <B>timeColumn</B>, and <B>summaryKeyColumn</B> fields must also be supplied. |
enabled | false | Boolean flag specifying whether the policy is to be enabled or not. |
filter | false | The Lucene filter specifying which events are to be synced. NOTE: If the policy is associated with a report, this cannot be changed - whatever is specified here will be ignored. |
maxBatchSize | false | The maximum number of events to write to the destination database in a single transaction. NOTE: A transaction may exceed this size if it is required to sync all of the events in a particular time boundary in a single transaction. |
maxEPSSize | false | The maximum number of events per second to sync. (NOTE: This is not currently used) |
partitionTable | false | A flag indicating whether the data should be stored in table partitions. If true, a new table partition will be created every day. Partitioning of the table makes deleting of old data much quicker, but it may also make queries against the table run slower. If a data sync policy is changed from partitioned to NOT partitioned, it is necessary to drop partitions and repopulate the data. After the partitions are dropped, a resync is initiated to cause this to happen. Similarly, if a data sync policy is changed from NOT partitioned to partitioned, it is necessary to truncate the data in the table so that the data can be populated into partitions. After the table is truncated, and partitioning enabled, a resync is initiated to cause this to happen. NOTE: This field is ignored if the data sync policy is NOT associated with a report data definition plugin. In other words, it is only used for data sync policies that are syncing data to the internal Postgres database. |
policyName | false | Name of data sync policy. NOTE: Policy names are required to be unique. If this is not a unique policy name, the update will fail. |
retentionPeriod | false | The number of days to retain data before deleting it. If missing, or if it contains a value <= 0, data is not deleted. |
ScheduleItems | false | If the <B>alwaysSchedule</B> field is "false", this is a nested JSON object that specifies the schedule to use for doing data sync. Note that it contains a single internal field <B>scheduleItem</B> that is an array of <B><Schedule Item></B> objects. See below for description of fields in Schedule Item objects. |
summaryKeyColumn | false | Column in destination table that holds a special summary key. The type of this column should be a VARCHAR capable of storing a 36 character UUID. This column is used internally when it is necessary to update a summary record. NOTE: This column should be indexed for performance reasons. |
summaryPeriod | false | Number of minutes to summarize events over. All events having a common set of event fields (as specified in the <B>TableColumnMap</B> field) will be counted over time periods of this length. A single record with a count of events found during the time period will be stored in the destination table. NOTE: This must be a positive number. If omitted, or the value is <= 0, the policy will be treated as a normal data sync policy - i.e., it will NOT produce summaries. |
syncInternalEvents | false | Boolean flag indicating whether or not to sync internal events. |
table | false | This is a nested JSON <B><Table></B> object that specifies the destination table events are to be synced to. See below for description of fields in Table objects. NOTE: If the policy is associated with a report, this cannot be changed - whatever is specified here will be ignored. |
TableColumnMap | false | This is a nested JSON object that specifies the mappings between event fields and destination table columns. Note that it contains a single internal field <B>ColumnMap</B> that is an array of <B><Column Map></B> objects. See below for description of fields in Column Map objects. NOTE: If the policy is associated with a report, this cannot be changed - whatever is specified here will be ignored. |
timeColumn | false | Column in destination table where event time will be stored. Event time for a summary record is defined to be the time at the beginning of the summary period. For example, if the summary period is two minutes, then event times would potentially fall on every two minute boundary (such as 12:00, 12:02, 12:04, etc). The count would be the count of all events which occurred starting from that time for the duration of the summary. If the time column contained a time of 12:02, then the summary record contains a count is for all events that occurred between >= 12:02 and < 12:04 (note it is exclusive of 12:04). NOTE: This column should be indexed for performance reasons. |
Object type: Data sync schedule item | ||
---|---|---|
Information about the data sync schedule for this policy | ||
Field | Required | Description |
dayOfWeek | false | Day of the week data sync should occur in. 0=Sunday, 1=Monday, etc. -1=Every Day |
duration | false | Number of minutes the data sync should last. 1 through 1440 (number of minutes in a day) |
startHour | false | Hour of the day the data sync should start. 0 through 23. |
startMinute | false | Minute of the hour the data sync should start. 0 through 59. |
Object type: Database connection object | ||
---|---|---|
Information about the connection to the database where the data sync data is to be stored | ||
Field | Required | Description |
database | false | Name of database. |
dbPlatform | false | Type of database. Valid values are: "postgresql", "oracle11g", and "mssql2008". |
hostName | false | Name or IP address of host where database resides. |
password | false | Password of database user. |
port | false | Port number for communication with database system. |
userName | false | User name of database user to login to database. |
Object type: Database table object | ||
---|---|---|
Information about the database table where the data sync data is to be stored | ||
Field | Required | Description |
schemaName | false | Name of the schema for the destination table. NOTE: This is an optional field. It will default to the schema of the database user specified in the database connection information. |
tableName | false | Name of the destination table. |
Object type: Data sync column mapping object | ||
---|---|---|
Information about how the lucene fields map to the database fields | ||
Field | Required | Description |
columnName | false | Name of column in the database that the event field is to be stored in. |
columnSize | false | Size of database column. NOTE: This only applies if the database column is a VARCHAR. |
columnType | false | Data type of database column. Should be a java.sql.Types value (BIGINT, VARCHAR, etc.). |
eventField | false | Name of event field that is to be synced. NOTE: These are the names of the event fields as specified <a target="_top" href="http://www.novell.com/developer/event_schema.html">here</a> in the <B>Tag Name</B> column. |
nullable | false | Flag indicating whether database column can have null values. 0=Nulls not allowed, 1=Nulls allowed, 2=Unknown if nulls allowed. |
Object type: Data sync policy object | |
---|---|
Information about the data sync policy | |
Field | Description |
alwaysSchedule | Boolean flag specifying whether data syncing is to run continuously ("true") or according to a schedule ("false"). If "false", the ScheduleItems field specifies the schedule to be used for syncing data. |
backOffPeriod | The number of seconds to backoff between queries when the synced-to time is up to date. This should be the value that was specified for the update request. |
countColumn | Column in destination table where event counts are to be stored. The type of this column should be capable of storing an integer. |
dbConnectionConfig | This is a nested JSON <B><Database Connection></B> object that specifies the destination database to sync data to. It is used in conjunction with the <B>table</B> field to specify the destination table events are to be synced to. See above for description of fields in Database Connection objects. |
doSummaries | Boolean flag specifying whether this policy creates event summary records instead of syncing individual events. If this field is not present, it defaults to false - i.e., the policy is NOT a summary policy. If true, then the <B>summaryPeriod</B>, <B>countColumn</B>, <B>timeColumn</B>, and <B>summaryKeyColumn</B> fields must also be supplied. |
enabled | Boolean flag specifying whether the policy is enabled or not. NOTE: It is possible for a policy to NOT be enabled even if the REST request specified that it should be. The policy may not be enabled if the destination database or destination table cannot be accessed. |
filter | The Lucene filter specifying which events are to be synced. |
forReporting | This is a flag that indicates whether the data sync policy is associated with a report. NOTE: This REST api is NOT allowed to change this field or the other fields that indicate what report the policy may be associated. However, the response data will return information indicating if the policy is associated with a report. |
id | This is the UUID of the data sync policy |
maxBatchSize | The maximum number of events to write to the destination database in a single transaction. This should be the value that was specified for the update request. NOTE: A transaction may exceed this size if it is required to sync all of the events in a particular time boundary in a single transaction. |
maxEPSSize | The maximum number of events per second to sync. This should be the value that was specified for the update request. |
partitionTable | A flag indicating whether the table is being partitioned or not. This should be the value that was specified for the update request. NOTE: This is only used for data sync policies that sync data to the internal Postgres database. |
policyName | Name of data sync policy. |
reportPluginDescription | This is the description of the report plugin the data sync policy is associated with. It will only be present if the <B>forReporting</B> field is "true". |
reportPluginDisplayName | This is the display name of the report plugin the data sync policy is associated with. It will only be present if the <B>forReporting</B> field is "true". |
reportPluginName | This is the name of the report plugin the data sync policy is associated with. It will only be present if the <B>forReporting</B> field is "true". |
reportPluginReleaseDate | This is the release date of the report plugin the data sync policy is associated with. It will only be present if the <B>forReporting</B> field is "true". |
retentionPeriod | The number of days to retain data before deleting it. This should be the value that was specified for the update request. |
ScheduleItems | If the <B>alwaysSchedule</B> field is "false", this is a nested JSON object that specifies the schedule to use for doing data sync. Note that it contains a single internal field <B>scheduleItem</B> that is an array of <B><Schedule Item></B> objects. See above for description of fields in Schedule Item objects. This should be whatever was specified for the update request. |
startSyncTime | This is the time that the data sync will be started from. It is specified in milliseconds since midnight, January 1, 1970 (UTC). |
summaryKeyColumn | Column in destination table that holds a special summary key. The type of this column should be a VARCHAR capable of storing a 36 character UUID. This column is used internally when it is necessary to update a summary record. NOTE: This column should be indexed for performance reasons. |
summaryPeriod | Number of minutes to summarize events over. All events having a common set of event fields (as specified in the <B>TableColumnMap</B> field) will be counted over time periods of this length. A single record with a count of events found during the time period will be stored in the destination table. NOTE: This must be a positive number. If omitted, or the value is <= 0, the policy will be treated as a normal data sync policy - i.e., it will NOT produce summaries. |
syncInternalEvents | Boolean flag indicating whether or not to sync internal events. |
table | This is a nested JSON <B><Table></B> object that specifies the destination table events are to be synced to. See below for description of fields in Table objects. |
TableColumnMap | This is a nested JSON object that specifies the mappings between event fields and destination table columns. Note that it contains a single internal field <B>ColumnMap</B> that is an array of <B><Column Map></B> objects. See above for description of fields in Column Map objects. |
timeColumn | Column in destination table where event time will be stored. Event time for a summary record is defined to be the time at the beginning of the summary period. For example, if the summary period is two minutes, then event times would potentially fall on every two minute boundary (such as 12:00, 12:02, 12:04, etc). The count would be the count of all events which occurred starting from that time for the duration of the summary. If the time column contained a time of 12:02, then the summary record contains a count is for all events that occurred between >= 12:02 and < 12:04 (note it is exclusive of 12:04). NOTE: This column should be indexed for performance reasons. |
PUT /datasync/policy/102B21D0-BE9B-102D-83DB-001A6B6D3CF6
Data Sync Policy Fields { "policyName": "My Data Sync Policy", "enabled": "true", "filter": "sev:[3 TO 5]", "syncInternalEvents": "false", "lagTime": "10", "retentionPeriod": "90", "partitionTable": "false", "backOffPeriod": "60", "maxEPSSize": "1000", "maxBatchSize": "100", "alwaysSchedule": "false", "ScheduleItems": { "scheduleItem": [{<Schedule Item>},{<Schedule Item>}...]}, "dbConnectionConfig":{<Database Connection>}, "table": {<Table>}, "fieldMappingStatus": {<Field Mapping Status>}, "TableColumnMap": { "ColumnMap": [{<Column Map>},{<Column Map>} ....]} "doSummaries": "false", "summaryPeriod": "0", "countColumn": "summary_count", "timeColumn": "summary_time", "summaryKeyColumn": "summary_key", "startSyncTime": "1288177541000", "forReporting": "false" } Schedule Item Fields { "dayOfWeek": "0", "startHour": "11", "startMinute": "23", "duration": "120" } Database Connection Fields { "hostName": "164.99.19.125", "port": "5432", "database": "SIEM", "userName": "appuser", "password": "star1111", "dbPlatform": "postgresql", } Table Fields { "schemaName": "my_schema", "tableName": "my_event_table" } Column Map Fields { "eventField": "msg", "columnName": "msg", "columnType": "12", "nullable": "1", "columnSize": "4000" }
{ "id": "102B21D0-BE9B-102D-83DB-001A6B6D3CF6", "policyName": "My Data Sync Policy", "enabled": "true", "filter": "sev:[3 TO 5]", "syncInternalEvents": "false", "lagTime": "10", "retentionPeriod": "90", "partitionTable": "false", "backOffPeriod": "60", "maxEPSSize": "1000", "maxBatchSize": "100", "alwaysSchedule": "false", "ScheduleItems": { "scheduleItem": [{<Schedule Item>},{<Schedule Item>}...]}, "dbConnectionConfig":{<Database Connection>}, "table": {<Table>}, "fieldMappingStatus": {<Field Mapping Status>}, "TableColumnMap": { "ColumnMap": [{<Column Map>},{<Column Map>} ....]} "doSummaries": "false", "summaryPeriod": "0", "countColumn": "summary_count", "timeColumn": "summary_time", "summaryKeyColumn": "summary_key", "startSyncTime": "1288177541000", "forReporting": "false" }