Introduction

With the release 1.2.0 of MRules, when lots of internal optimizations were brought and some key functionalities introduced (such as strict access to fields and methods), we decided to realize a benchmark of our product, comparing it to other actors on the market.

The opponent is a major one: it’s Drools, the Open Source leader of rule engines. Based on the RETE algorithm, this library has the reputation to offer great performances.

For this test, a fictive but realistic business case has been elaborated, the Object model implemented, the rules written and so where the datasets.

Business case description

We have imagined the case of a laboratory, analyzing air quality. This laboratory thus receives regularly measure reports, each one composed by several samples and concerning one or many pollutants.

When receiving a measure report, data consistency is checked, according to a set of business rules. This verification may raise errors, blocking or not.

Following validation, if no blocking error is raised, an air quality index is computed.

Data description
A measure is composed with the following data:

• Start and end dates
• A location
• Contacts (emails)
• A number of samples
• A sample frequency (Minute, Hour, Day)
• Sample data, represented as a Map (Key / Values association) :
  • Key: pollutant type
  • Value: Java object holding pollutant type and measured value

The last data modelization allows to complexify the model and to test the products’ abilities and configurations.
An error is represented with a Java Object:

• Rule code
• Error message
• Blocking or not

Rules description

All implemented rules are described bellow:
Code “DatesOk”:

• Start and end dates can’t be empty (Blocking).
• End date can’t be before start date (Blocking).

Code “LocationOk”:

• Location can’t be empty (Blocking).

Code “ContactsOk”:

• At least one contact is present (Warning).
• Contacts must be valid emails. Check is done using regexp (Warning).

Code “SamplesOk”:

• At least 1 sample; at most 10 (Blocking).
• Frequency must be filled with a valid value (in M, H, D) (Blocking).
• At least 1 pollutant is measured, at most 5 (Blocking).
• Pollutant type must be known (i.e. present in reference data outside rules) (Blocking).

Finalize:

Rules computing output data:

• Computing the total number of blocking errors.
• Then, if no blockers:
  • If one pollutant measured over the threshold of 0.02: “BAD” air quality.
  • If one pollutant measured over the threshold of 0.01: “MEDIUM” air quality.
  • Otherwise: “GOOD” air quality..

Test cases

16 test cases have been created, some with one or more error(s), others without. The objective was to test all defined rules.

Test implementation
The benchmark was written as a Maven Project and JUnit tests.
3 UT have been written:

• One executing native Java rules.
• One executing Drools.
• One executing MRules.

Drools configuration has been written using DRL language. Excel decision table format has been tried, but some of the rules never ran by this way.

Here is the final DRL configuration.

Mrules configuration was written in XML format.

You can also download the full bench mark project (a valid licence is required).

Résultats du test

The goal in this benchmark is to compare two main points:

• Integration
• Configuration
• Performances, computed on a standard computer for 1 compilation and 10,000 runs of the 16 cases. For MRules, tests were launched in both standard and light compilation modes.

* Figures will vary from one computer to another depending on the available computing power. What is interesting here are the proportions.

Interpretation

For integration

The two libraries are very similar: few dependencies and few Java code to write. However, it’s easy to write a utility centralizing rule engines usages. Drools has more transitive dependencies and needs more space.

For configuration

With the Java implementation, all options are available, depending on the budget.

With Drools, using the DRL language, code is replaced … by code. Moreover, Drools ignores the rules’ ordering in the DRL file, so it’s necessary to add specific instructions to refresh data or avoid infinite loops. A motivated non-technical contributor will be able to modify threshold values, but certainly not to change the behavior.

With MRules, the XML language allows rules to be written in a mechanical and supervised way. A non-technical contributor will be able to understand how it works, modify threshold values and change some behaviors. For complex changes, a technical support may be necessary.

(Please note that we currently work on other configuration formats – Excel, text – allowing addressing all final user types).

For performances

Here, raw results speak for themselves.

Java implementation is faster for this type of rules: sequential, limited in number, with few or no algorithm and strongly business oriented. Indeed, no abstraction layer is induced, provoking an additional processing cost.

On the other hand, our efforts in terms of design and optimization for the targeted use cases are paying. In this niche and this type of business, our engine is significantly better than Drools in terms of performance, both in compilation and execution phases.

We are working hard for future versions on new algorithms and optimizations, in order to get closer to the reference time of the Java implementation … or even bellow.