The benchmarker is in a separate artifact called optaplanner-benchmark.

If you use Maven, add a dependency in your pom.xml file:

    <dependency>

      <groupId>org.optaplanner</groupId>

      <artifactId>optaplanner-benchmark</artifactId>

      <version>...</version>

    </dependency>

This is similar for Gradle, Ivy and Buildr. The version must be exactly the same as the optaplanner-core version used.

If you use ANT, you've probably already copied the required jars from the download zip's binaries directory.

You can build a PlannerBenchmark instance with the XmlPlannerBenchmarkFactory. Configure it with a benchmark configuration xml file:

        PlannerBenchmarkFactory plannerBenchmarkFactory = new XmlPlannerBenchmarkFactory(

                "/org/optaplanner/examples/nqueens/benchmark/nqueensBenchmarkConfig.xml");

        PlannerBenchmark plannerBenchmark = benchmarkFactory.buildPlannerBenchmark();

        plannerBenchmark.benchmark();

A basic benchmark configuration file looks something like this:

<?xml version="1.0" encoding="UTF-8"?>

<plannerBenchmark>

  <benchmarkDirectory>local/data/nqueens</benchmarkDirectory>

  <!--<parallelBenchmarkCount>AUTO</parallelBenchmarkCount>-->

  <warmUpSecondsSpend>30</warmUpSecondsSpend>



  <inheritedSolverBenchmark>

    <problemBenchmarks>

      <xstreamAnnotatedClass>org.optaplanner.examples.nqueens.domain.NQueens</xstreamAnnotatedClass>

      <inputSolutionFile>data/nqueens/unsolved/unsolvedNQueens32.xml</inputSolutionFile>

      <inputSolutionFile>data/nqueens/unsolved/unsolvedNQueens64.xml</inputSolutionFile>

      <problemStatisticType>BEST_SOLUTION_CHANGED</problemStatisticType>

    </problemBenchmarks>

    <solver>

      <solutionClass>org.optaplanner.examples.nqueens.domain.NQueens</solutionClass>

      <planningEntityClass>org.optaplanner.examples.nqueens.domain.Queen</planningEntityClass>

      <scoreDirectorFactory>

        <scoreDefinitionType>SIMPLE</scoreDefinitionType>

        <scoreDrl>/org/optaplanner/examples/nqueens/solver/nQueensScoreRules.drl</scoreDrl>

      </scoreDirectorFactory>

      <termination>

        <maximumSecondsSpend>20</maximumSecondsSpend>

      </termination>

      <constructionHeuristic>

        <constructionHeuristicType>FIRST_FIT_DECREASING</constructionHeuristicType>

        <constructionHeuristicPickEarlyType>FIRST_LAST_STEP_SCORE_EQUAL_OR_IMPROVING</constructionHeuristicPickEarlyType>

      </constructionHeuristic>

    </solver>

  </inheritedSolverBenchmark>



  <solverBenchmark>

    <name>Entity tabu</name>

    <solver>

      <localSearch>

        <changeMoveSelector>

          <selectionOrder>ORIGINAL</selectionOrder>

        </changeMoveSelector>

        <acceptor>

          <entityTabuSize>5</entityTabuSize>

        </acceptor>

        <forager>

          <pickEarlyType>NEVER</pickEarlyType>

        </forager>

      </localSearch>

    </solver>

  </solverBenchmark>

  <solverBenchmark>

    <name>Value tabu</name>

    <solver>

      <localSearch>

        <changeMoveSelector>

          <selectionOrder>ORIGINAL</selectionOrder>

        </changeMoveSelector>

        <acceptor>

          <valueTabuSize>5</valueTabuSize>

        </acceptor>

        <forager>

          <pickEarlyType>NEVER</pickEarlyType>

        </forager>

      </localSearch>

    </solver>

  </solverBenchmark>

  <solverBenchmark>

    <name>Move tabu</name>

    <solver>

      <localSearch>

        <changeMoveSelector>

          <selectionOrder>ORIGINAL</selectionOrder>

        </changeMoveSelector>

        <acceptor>

          <moveTabuSize>5</moveTabuSize>

        </acceptor>

        <forager>

          <pickEarlyType>NEVER</pickEarlyType>

        </forager>

      </localSearch>

    </solver>

  </solverBenchmark>

</plannerBenchmark>

This PlannerBenchmark will try 3 configurations (1 move tabu, 1 entity tabu and 1 value tabu) on 2 data sets (32 and 64 queens), so it will run 6 solvers.

Every solverBenchmark element contains a solver configuration (for example with a local search solver phase) and one or more inputSolutionFile elements. It will run the solver configuration on each of those unsolved solution files. The element name is optional, because it is generated if absent. The inputSolutionFile is read by a ProblemIO.

To lower verbosity, the common part of multiple solverBenchmark entities can be extracted to the inheritedSolverBenchmark element. Yet, every element can still be overwritten per solverBenchmark element. Note that inherited solver phases such as <constructionHeuristic> or <localSearch> are not overwritten but instead are added to the tail of the solver phases list.

You need to specify a benchmarkDirectory (relative to the working directory). A benchmark report will be written in that directory.

The best solution of each benchmark run can be written to the in the benchmarkDirectory. By default, this is disabled, because the files are rarely used and considered bloat. Also, on large datasets, writing the best solution of each single benchmark can take quite some time and memory (causing an OutOfMemoryError), especially in a verbose format like XStream.

You can enable to write the output solution in the benchmarkDirectory with writeOutputSolutionEnabled:

    <problemBenchmarks>

      ...

      <writeOutputSolutionEnabled>true</writeOutputSolutionEnabled>

      ...

    </problemBenchmarks>

Without a warm up, the results of the first (or first few) benchmarks are not reliable, because they will have lost CPU time on HotSpot JIT compilation (and possibly DRL compilation too).

The avoid that distortion, the benchmarker can run some of the benchmarks for a specified amount of time, before running the real benchmarks. Generally, a warm up of 30 seconds suffices:

<plannerBenchmark>

  ...

  <warmUpSecondsSpend>30</warmUpSecondsSpend>

  ...

</plannerBenchmark>

After the running a benchmark, a HTML report will be written in the benchmarkDirectory with the filename index.html. Open it in your browser. It has a nice overview of your benchmark including:

The HTML report will use your default locale to format numbers. If you need to share the benchmark report with people from another country, you might want to overwrite the benchmarkReportLocale:

<plannerBenchmark>

  ...

  <benchmarkReportLocale>en_US</benchmarkReportLocale>

  ...

</plannerBenchmark>

The benchmark report automatically ranks the solvers. The Solver with rank 0 is called the favorite Solver: it performs best overall, but it might not be the best on every problem. It's recommended to use that favorite Solver in production.

However, there are different ways of ranking the solvers. You can configure how:

<plannerBenchmark>

  ...

  <solverBenchmarkRankingType>TOTAL_SCORE</solverBenchmarkRankingType>

  ...

</plannerBenchmark>

The following solverBenchmarkRankingTypes are supported:

You can also use a custom ranking, by implementing a Comparator:

  <solverBenchmarkRankingComparatorClass>...TotalScoreSolverBenchmarkRankingComparator</solverBenchmarkRankingComparatorClass>

Or a weight factory:

  <solverBenchmarkRankingWeightFactoryClass>...TotalRankSolverBenchmarkRankingWeightFactory</solverBenchmarkRankingWeightFactoryClass>

Matrix benchmarking is benchmarking a combination of value sets. For example: benchmark 4 entityTabuSize values (5, 7, 11 and 13) combined with 3 acceptedCountLimit values (500, 1000 and 2000), resulting in 12 solver configurations.

To reduce the verbosity of such a benchmark configuration, you can use a Freemarker template for the benchmark configuration instead:

<plannerBenchmark>

  ...



  <inheritedSolverBenchmark>

    ...

  </inheritedSolverBenchmark>



<#list [5, 7, 11, 13] as entityTabuSize>

<#list [500, 1000, 2000] as acceptedCountLimit>

  <solverBenchmark>

    <name>entityTabuSize ${entityTabuSize} acceptedCountLimit ${acceptedCountLimit}</name>

    <solver>

      <localSearch>

        <unionMoveSelector>

          <changeMoveSelector/>

          <swapMoveSelector/>

        </unionMoveSelector>

        <acceptor>

          <entityTabuSize>${entityTabuSize}</entityTabuSize>

        </acceptor>

        <forager>

          <acceptedCountLimit>${acceptedCountLimit}</acceptedCountLimit>

        </forager>

      </localSearch>

    </solver>

  </solverBenchmark>

</#list>

</#list>

</plannerBenchmark>

And build it with the class FreemarkerXmlPlannerBenchmarkFactory:

        PlannerBenchmarkFactory plannerBenchmarkFactory = new FreemarkerXmlPlannerBenchmarkFactory(

                "/org/optaplanner/examples/cloudbalancing/benchmark/cloudBalancingBenchmarkConfigTemplate.xml.ftl");

        PlannerBenchmark plannerBenchmark = benchmarkFactory.buildPlannerBenchmark();