Performance & Profiling script¶

Note

The performance script got its own Sphinx extension: Sphinx-Performance.

This extension is based on the described script, but is more powerfull and better maintained. It can also be used for general performance analysis of Sphinx and its extensions.

The performance of Sphinx-Needs can be tested by a script called performance_test.py inside folder /performance of the checked out github repository.

The performance can be tested with different amounts of needs, needtables and not Sphinx-Needs related dummies (simple rst code).

Test series¶

To start a series of test with some predefined values, run python performance_test.py series

Running 8 test configurations.
* Running on 5 pages with 50 needs, 5 needtables, 5 dummies per page. Using 1 cores.
  Duration: 8.05 seconds
* Running on 5 pages with 50 needs, 5 needtables, 5 dummies per page. Using 4 cores.
  Duration: 6.75 seconds
* Running on 1 pages with 50 needs, 5 needtables, 5 dummies per page. Using 1 cores.
  Duration: 2.16 seconds
* Running on 1 pages with 50 needs, 5 needtables, 5 dummies per page. Using 4 cores.
  Duration: 2.36 seconds
* Running on 5 pages with 10 needs, 1 needtables, 1 dummies per page. Using 1 cores.
  Duration: 2.39 seconds
* Running on 5 pages with 10 needs, 1 needtables, 1 dummies per page. Using 4 cores.
  Duration: 2.34 seconds
* Running on 1 pages with 10 needs, 1 needtables, 1 dummies per page. Using 1 cores.
  Duration: 1.69 seconds
* Running on 1 pages with 10 needs, 1 needtables, 1 dummies per page. Using 4 cores.
  Duration: 1.70 seconds

RESULTS:

  runtime      pages       needs      needs    needtables    dummies    parallel
  seconds    overall    per page    overall       overall    overall       cores
---------  ---------  ----------  ---------  ------------  ---------  ----------
     8.05          5          50        250            25         25           1
     6.75          5          50        250            25         25           4
     2.16          1          50         50             5          5           1
     2.36          1          50         50             5          5           4
     2.39          5          10         50             5          5           1
     2.34          5          10         50             5          5           4
     1.69          1          10         10             1          1           1
     1.7           1          10         10             1          1           4

Overall runtime: 27.45 seconds.

But you can modify the details and set some static values by setting various parameters. Just run python performance_test.py series --help to get an overview

Usage: performance_test.py series [OPTIONS]

  Generate and start a series of tests.

Options:
  --profile TEXT        Activates profiling for given area
  --needs INTEGER       Number of maximum needs.
  --needtables INTEGER  Number of maximum needtables.
  --dummies INTEGER     Number of standard rst dummies.
  --pages INTEGER       Number of additional pages with needs.
  --parallel INTEGER    Number of parallel processes to use. Same as -j for
                        sphinx-build
  --keep                Keeps the temporary src and build folders
  --browser             Opens the project in your browser
  --snakeviz            Opens snakeviz view for measured profiles in browser
  --debug               Prints more information, incl. sphinx build output
  --basic               Use only default config of Sphinx-Needs (e.g. no extra
                        options)
  --help                Show this message and exit.

Also if --needs, --pages or parallel is set multiple times, one performance test is executed per it.

Example:: python performance_test.py series --needs 1 --needs 10 --pages 1 --pages 10 --parallel 1 --parallel 4 --needtables 0 --dummies 0. This will set 2 values for needs, 2 for pages and 2 for parallel. So in the end it will run 8 test configurations (2 needs x 2 pages x 2 parallel = 8).

Running 8 test configurations.
* Running on 1 pages with 1 needs, 0 needtables, 0 dummies per page. Using 1 cores.
  Duration: 1.53 seconds
* Running on 1 pages with 1 needs, 0 needtables, 0 dummies per page. Using 4 cores.
  Duration: 1.64 seconds
* Running on 10 pages with 1 needs, 0 needtables, 0 dummies per page. Using 1 cores.
  Duration: 1.96 seconds
* Running on 10 pages with 1 needs, 0 needtables, 0 dummies per page. Using 4 cores.
  Duration: 2.01 seconds
* Running on 1 pages with 10 needs, 0 needtables, 0 dummies per page. Using 1 cores.
  Duration: 1.91 seconds
* Running on 1 pages with 10 needs, 0 needtables, 0 dummies per page. Using 4 cores.
  Duration: 1.93 seconds
* Running on 10 pages with 10 needs, 0 needtables, 0 dummies per page. Using 1 cores.
  Duration: 2.94 seconds
* Running on 10 pages with 10 needs, 0 needtables, 0 dummies per page. Using 4 cores.
  Duration: 2.48 seconds

RESULTS:

  runtime      pages       needs      needs    needtables    dummies    parallel
  seconds    overall    per page    overall       overall    overall       cores
---------  ---------  ----------  ---------  ------------  ---------  ----------
     1.53          1           1          1             0          0           1
     1.64          1           1          1             0          0           4
     1.96         10           1         10             0          0           1
     2.01         10           1         10             0          0           4
     1.91          1          10         10             0          0           1
     1.93          1          10         10             0          0           4
     2.94         10          10        100             0          0           1
     2.48         10          10        100             0          0           4

Overall runtime: 16.41 seconds.

Parallel execution¶

versionadded:: 0.7.1

You may have noticed, the parallel execution on multiple cores can lower the needed runtime.

This parallel execution is using the “-j” option from sphinx-build. This mostly brings benefit, if dozens/hundreds of files need to be read and written. In this case sphinx starts several workers to deal with these files in parallel.

If the project contains only a few files, the benefit is not really measurable.

Here an example of a 500 page project, build once on 1 and 8 cores. The benefit is ~40% of build time, if 8 cores are used.

  runtime s    pages #    needs per page    needs #    needtables #    dummies #    parallel cores
-----------  ---------  ----------------  ---------  --------------  -----------  ----------------
     169.46        500                10       5000               0         5000                 1
     103.08        500                10       5000               0         5000                 8

Used command: python performance_test.py series --needs 10 --pages 500 --dummies 10 --needtables 0 --parallel 1 --parallel 8

The parallel execution can used by any documentation build , just use -j option. Example, which uses 4 processes in parallel: sphinx-build -j 4 -b html . _build/html

Used rst template¶

For all performance tests the same rst-template is used:

index¶

Performance test
================

Config
------
:dummies: {{dummies}}
:needs: {{needs}}
:needtables: {{needtables}}
:keep: {{keep}}
:browser: {{browser}}
:debug: {{debug}}

Content
-------
.. contents::

.. toctree::

{% for page in range(pages) %}
   page_{{page}}
{% endfor -%}

pages¶

{{ title}}
{{ "=" * title|length }}

Test Data
---------

Dummies
~~~~~~~
Amount of dummies: **{{dummies}}**

{% for n in range(dummies) %}
**Dummy {{n}}**

.. note::  This is dummy {{n}}

And some **dummy** *text* for dummy {{n}}

{% endfor %}

Needs
~~~~~
Amount of needs: **{{needs}}**

{% for n in range(needs) %}
.. req:: Test Need Page {{ page }} {{n}}
   :id: R_{{page}}_{{n}}
{% if not basic %}   :number: {{n}}{% endif %}
   :links: R_{{page}}_{{needs-n-1}}
{% endfor %}

Needtable
~~~~~~~~~
Amount of needtables: **{{needtables}}**

{%  if basic %}
.. needtable::
   :show_filters:
   :columns: id, title, number, links
{% else %}
{% for n in range(needtables) %}
.. needtable::
   :show_filters:
   :filter: int(number)**3 > 0 or len(links) > 0
   :columns: id, title, number, links
{% endfor %}
{% endif %}

Profiling¶

With option --profile NAME a code-area specific profile can be activated.

Currently supported are:

NEEDTABLE: Profiles the needtable processing (incl. printing)
NEED_PROCESS: Profiles the need processing (without printing)
NEED_PRINT: Profiles the need painting (creating final nodes)

If this option is used, a profile folder gets created in the current working directory and a profile file with <NAME>.prof is created. This file contains CProfile Stats information.

--profile can be used several times.

These profile can be also created outside the performance test with each documentation project. Simply set a environment variable called NEEDS_PROFILING and set the value to the needed profiles.

Example for Linux: export NEEDS_PROFILING=NEEDTABLE,NEED_PRINT.

Analysing profile¶

Use snakeviz together with --profile <NAME> to open automatically a graphical analysis of the generated profile file.

For this snakeviz must be installed: pip install snakeviz.

Example:

python performance_test.py series --needs 10 --pages 10 --profile NEEDTABLE --profile NEED_PROCESS --snakeviz

Measurements¶

The measurements were performed with the following setup:

Sphinx-Needs 0.7.0 on 1 core as parallel build is not supported by version.
Sphinx-Needs 0.7.1, with 1 core.
Sphinx-Needs 0.7.1, with 4 cores.

Test details	0.7.0 with 1 core	0.7.1 with 1 core	0.7.1 with 4 cores
30 pages with overall 1500 needs and 30 needtables	55.02 s	36.81 s	34.31 s
100 pages with overall 10.000 needs and 100 needtables	6108.26 s	728.82 s	564.76 s