In the context of streaming events from devices there is a need for extracting information based on a predefined set of rules. Rules are defined per device and user. They are also divided in two categories based on the condition under which they are triggered. Filtering conditions are triggered when a set of filters is fulfilled. Triggering conditions use filters to set up the rule body, only they are triggered based on conditions set up by aggregated values, e.g. total, min, max, occurrence, average. The solution needs to adapt well to situations when the throughput of input stream of events becomes high (scalability).
The original ruleset implementation was proved to not scale well. The set of filtering/ triggering conditions, being essentially a tree, was evaluated from root to leafs using recursion to address continuous loops of evaluation and Boolean functions. The evaluation happened for every event, for each rule, leading to events “piled up” at the entrance of the algorithm. Therefore, a different approach was needed in order to achieve better scaling.
The main idea of the algorithm is to enable the evaluation of the conditions tree in a single pass. We start by retrieving the list of devices and then, for each one, we parse the rules that are referring to it, building the tree in memory. The key feature is that we plan ahead to evaluate the tree from leafs to root in a single pass. The relevant structures, are built in such a way that become ready to enter as input to the evaluation function. The evaluation function is a simple neural network (NN), that uses weights to implement basic Boolean functions.
The process described above has been implemented through a new module named “ruleflow”. The activation function is a sigmoid function ( f(x) = 1 / (1+ e^(-x)) ). The input can be described as a vector, the same stands true for the weights.
Condition tree evaluation is very fast, due to the absence of recursions at parsing of stream in real time. Recursion happens in tree building in the process of caching every 5 minutes as a concurrent procedure to the stream processor in order to update rules info per device, but certainly not per event. Also, splitting the input stream evaluation in multiple evaluation modules has proven to scale well.
Filtering and triggering conditions are treated in the same manner due to the nature of the algorithm. Triggering is built as a module on top of filtering, aggregating values in order to fulfill the rule. There is some room for improvement, since all events from multiple parallel processes are updating the same metrics database. In rare cases, this may also lead to inaccuracies since the same keys may be incremented at the same time. We plan to improve this process and eliminate any occasional inaccuracies by a map-reduce procedure between concurrent threads.