The benchmarker is in a separate artifact called optaplanner-benchmark.

If you use Maven, add a dependency in your pom.xml file:

    <dependency>

      <groupId>org.optaplanner</groupId>

      <artifactId>optaplanner-benchmark</artifactId>

    </dependency>

This is similar for Gradle, Ivy and Buildr. The version must be exactly the same as the optaplanner-core version used (which is automatically the case if you import optaplanner-bom).

If you use ANT, you've probably already copied the required jars from the download zip's binaries directory.

You can build a PlannerBenchmark instance with the XmlPlannerBenchmarkFactory. Configure it with a benchmark configuration xml file:

        PlannerBenchmarkFactory plannerBenchmarkFactory = new XmlPlannerBenchmarkFactory(

                "/org/optaplanner/examples/nqueens/benchmark/nqueensBenchmarkConfig.xml");

        PlannerBenchmark plannerBenchmark = benchmarkFactory.buildPlannerBenchmark();

        plannerBenchmark.benchmark();

A basic benchmark configuration file looks something like this:

<?xml version="1.0" encoding="UTF-8"?>

<plannerBenchmark>

  <benchmarkDirectory>local/data/nqueens</benchmarkDirectory>

  <!--<parallelBenchmarkCount>AUTO</parallelBenchmarkCount>-->

  <warmUpSecondsSpend>30</warmUpSecondsSpend>



  <inheritedSolverBenchmark>

    <problemBenchmarks>

      <xstreamAnnotatedClass>org.optaplanner.examples.nqueens.domain.NQueens</xstreamAnnotatedClass>

      <inputSolutionFile>data/nqueens/unsolved/32queens.xml</inputSolutionFile>

      <inputSolutionFile>data/nqueens/unsolved/64queens.xml</inputSolutionFile>

      <problemStatisticType>BEST_SCORE</problemStatisticType>

    </problemBenchmarks>

    <solver>

      <solutionClass>org.optaplanner.examples.nqueens.domain.NQueens</solutionClass>

      <planningEntityClass>org.optaplanner.examples.nqueens.domain.Queen</planningEntityClass>

      <scoreDirectorFactory>

        <scoreDefinitionType>SIMPLE</scoreDefinitionType>

        <scoreDrl>/org/optaplanner/examples/nqueens/solver/nQueensScoreRules.drl</scoreDrl>

      </scoreDirectorFactory>

      <termination>

        <maximumSecondsSpend>20</maximumSecondsSpend>

      </termination>

      <constructionHeuristic>

        <constructionHeuristicType>FIRST_FIT_DECREASING</constructionHeuristicType>

        <forager>

          <pickEarlyType>FIRST_NON_DETERIORATING_SCORE</pickEarlyType>

        </forager>

      </constructionHeuristic>

    </solver>

  </inheritedSolverBenchmark>



  <solverBenchmark>

    <name>Entity tabu</name>

    <solver>

      <localSearch>

        <changeMoveSelector>

          <selectionOrder>ORIGINAL</selectionOrder>

        </changeMoveSelector>

        <acceptor>

          <entityTabuSize>5</entityTabuSize>

        </acceptor>

        <forager>

          <pickEarlyType>NEVER</pickEarlyType>

        </forager>

      </localSearch>

    </solver>

  </solverBenchmark>

  <solverBenchmark>

    <name>Value tabu</name>

    <solver>

      <localSearch>

        <changeMoveSelector>

          <selectionOrder>ORIGINAL</selectionOrder>

        </changeMoveSelector>

        <acceptor>

          <valueTabuSize>5</valueTabuSize>

        </acceptor>

        <forager>

          <pickEarlyType>NEVER</pickEarlyType>

        </forager>

      </localSearch>

    </solver>

  </solverBenchmark>

  <solverBenchmark>

    <name>Move tabu</name>

    <solver>

      <localSearch>

        <changeMoveSelector>

          <selectionOrder>ORIGINAL</selectionOrder>

        </changeMoveSelector>

        <acceptor>

          <moveTabuSize>5</moveTabuSize>

        </acceptor>

        <forager>

          <pickEarlyType>NEVER</pickEarlyType>

        </forager>

      </localSearch>

    </solver>

  </solverBenchmark>

</plannerBenchmark>

This PlannerBenchmark will try 3 configurations (1 move tabu, 1 entity tabu and 1 value tabu) on 2 data sets (32 and 64 queens), so it will run 6 solvers.

Every <solverBenchmark> element contains a solver configuration (for example with a local search solver phase) and one or more <inputSolutionFile> elements. It will run the solver configuration on each of those unsolved solution files. The element name is optional, because it is generated if absent. The inputSolutionFile is read by a ProblemIO.

To lower verbosity, the common part of multiple <solverBenchmark> elements can be extracted to the <inheritedSolverBenchmark> element. Yet, every property can still be overwritten per <solverBenchmark> element. Note that inherited solver phases such as <constructionHeuristic> or <localSearch> are not overwritten but instead are added to the tail of the solver phases list.

You need to specify a <benchmarkDirectory> element (relative to the working directory). A benchmark report will be written in that directory.

If an Exception or Error occurs in a single benchmark, the entire Benchmarker will not fail-fast (unlike everything else in OptaPlanner). Instead, the Benchmarker will continue to run all other benchmarks, write the benchmark report and then fail (if there is at least 1 failing single benchmark). The failing benchmarks will be clearly marked in the benchmark report.

The best solution of each benchmark run can be written to the in the benchmarkDirectory. By default, this is disabled, because the files are rarely used and considered bloat. Also, on large datasets, writing the best solution of each single benchmark can take quite some time and memory (causing an OutOfMemoryError), especially in a verbose format like XStream.

You can enable to write the output solution in the benchmarkDirectory with writeOutputSolutionEnabled:

    <problemBenchmarks>

      ...

      <writeOutputSolutionEnabled>true</writeOutputSolutionEnabled>

      ...

    </problemBenchmarks>

Without a warm up, the results of the first (or first few) benchmarks are not reliable, because they will have lost CPU time on HotSpot JIT compilation (and possibly DRL compilation too).

The avoid that distortion, the benchmarker can run some of the benchmarks for a specified amount of time, before running the real benchmarks. Generally, a warm up of 30 seconds suffices:

<plannerBenchmark>

  ...

  <warmUpSecondsSpend>30</warmUpSecondsSpend>

  ...

</plannerBenchmark>

After the running a benchmark, a HTML report will be written in the benchmarkDirectory with the filename index.html. Open it in your browser. It has a nice overview of your benchmark including:

The HTML report will use your default locale to format numbers. If you need to share the benchmark report with people from another country, you might want to overwrite the benchmarkReportLocale:

<plannerBenchmark>

  ...

  <benchmarkReportLocale>en_US</benchmarkReportLocale>

  ...

</plannerBenchmark>

The benchmark report automatically ranks the solvers. The Solver with rank 0 is called the favorite Solver: it performs best overall, but it might not be the best on every problem. It's recommended to use that favorite Solver in production.

However, there are different ways of ranking the solvers. Configure it like this:

<plannerBenchmark>

  ...

  <solverBenchmarkRankingType>TOTAL_SCORE</solverBenchmarkRankingType>

  ...

</plannerBenchmark>

The following solverBenchmarkRankingTypes are supported:

You can also use a custom ranking, by implementing a Comparator:

  <solverBenchmarkRankingComparatorClass>...TotalScoreSolverBenchmarkRankingComparator</solverBenchmarkRankingComparatorClass>

Or a weight factory:

  <solverBenchmarkRankingWeightFactoryClass>...TotalRankSolverBenchmarkRankingWeightFactory</solverBenchmarkRankingWeightFactoryClass>

Matrix benchmarking is benchmarking a combination of value sets. For example: benchmark 4 entityTabuSize values (5, 7, 11 and 13) combined with 3 acceptedCountLimit values (500, 1000 and 2000), resulting in 12 solver configurations.

To reduce the verbosity of such a benchmark configuration, you can use a Freemarker template for the benchmark configuration instead:

<plannerBenchmark>

  ...



  <inheritedSolverBenchmark>

    ...

  </inheritedSolverBenchmark>



<#list [5, 7, 11, 13] as entityTabuSize>

<#list [500, 1000, 2000] as acceptedCountLimit>

  <solverBenchmark>

    <name>entityTabuSize ${entityTabuSize} acceptedCountLimit ${acceptedCountLimit}</name>

    <solver>

      <localSearch>

        <unionMoveSelector>

          <changeMoveSelector/>

          <swapMoveSelector/>

        </unionMoveSelector>

        <acceptor>

          <entityTabuSize>${entityTabuSize}</entityTabuSize>

        </acceptor>

        <forager>

          <acceptedCountLimit>${acceptedCountLimit}</acceptedCountLimit>

        </forager>

      </localSearch>

    </solver>

  </solverBenchmark>

</#list>

</#list>

</plannerBenchmark>

And build it with the class FreemarkerXmlPlannerBenchmarkFactory:

        PlannerBenchmarkFactory plannerBenchmarkFactory = new FreemarkerXmlPlannerBenchmarkFactory(

                "/org/optaplanner/examples/cloudbalancing/benchmark/cloudBalancingBenchmarkConfigTemplate.xml.ftl");

        PlannerBenchmark plannerBenchmark = benchmarkFactory.buildPlannerBenchmark();