Derived fields¶

Added in version 9.0.0.

Many values in a requirements project are not decided on the need that shows them, but follow from other needs: the effort of a story is the sum of the effort of its tasks, a requirement is verified once every test of it has passed, a test result is evidence for the requirements its test traces to. Written by hand, such a value is out of date as soon as one of those needs changes.

A field whose entry carries a derive table is derived: a rule computes its value for every need the project creates from its sources, or only for the needs of the need types it is declared on, and an author does not write it. The table names a kind, what to compute, and the kind’s operands, each under a fixed role name. In the TOML file:

[needs.links.tests]

[needs.fields.hours]
schema = { type = "number" }

[needs.fields.total_hours]
schema = { type = "number" }
derive = { kind = "sum", field = "hours", over = "links" }

[needs.fields.verified]
schema = { type = "boolean" }
derive = { kind = "all", test = "status == 'passed'", over = "tests_back", where = "type == 'test'", empty = false }

[needs.links.mentions]
derive = { kind = "content_links" }

total_hours adds up the hours of the needs a need links to, verified says whether the need has a test naming it in its tests and every such test has passed (empty = false makes a need no test names false, not “verified”), and mentions links the needs that a need’s text references. The total_hours rule in conf.py, as every other key of the entry:

needs_fields = {
    "total_hours": {
        "schema": {"type": "number"},
        "derive": {"kind": "sum", "field": "hours", "over": "links"},
    },
}

A rule for the needs of some need types is declared in those types’ own entries (see Rules per need type):

[needs.types.task]
title = "Task"
prefix = "T_"

[needs.types.story]
title = "Story"
prefix = "S_"

[needs.fields.effort]
schema = { type = "number" }

[needs.types.story.fields.total_effort]
schema = { type = "number" }
derive = { kind = "sum", field = "effort", over = "links" }

gives a story the sum of the effort of the needs it links, and a need of another type None.

A link type is derived the same way, with derive on its needs_links entry and one of the kinds that compute a link list: links, content_links and union, or copy and collect, which there link the far ends of a chain of other links (A chain as links: copy and collect on a link type). A derived field is closed to authors: a value written for it in a need is ignored, and reported (see What a derived field holds). How a rule relates to the [[…]] calls that compute a value per need is described in Rules and inline calls. Whether a value is better computed by a rule, an inline call or a variant is set out in Delayed field evaluation.

Before declaring a rule, bear in mind:

Nobody can write the value. A value written for a derived field in a need, or set by a needextend, is ignored and reported (needs.derive_authored). A value that someone must be able to write, or to correct by hand, needs a field of its own.
Most mistakes are found when the configuration is read. A rule that cannot be read is reported as needs.derive_invalid, and the needs it would compute hold the field’s empty value; one that cannot be computed on a need is reported there (What a derived field holds).
Storing costs, computing does not. A value per need costs little; a rule that links many needs to many stores a list on each need and a back link on each target (needs.derive_dense, What a rule costs).
Incremental builds re-write more pages. When a need changes, the pages of the needs whose rules read it are re-written too; once a need of the project holds a links rule, a content_links rule with from, a collect on a link type, or a copy on a link type over a link type a rule computes, every change of a need re-writes every page holding a need, a view or a need role (Incremental builds).
A simpler tool may do. To show a set of needs in one place, a needtable or needlist with a filter shows it once; a value for one need alone is a [[…]] call written in it (Rules and inline calls).

Declaring a rule¶

derive in a field’s needs_fields entry, or in a link type’s needs_links entry, gives its rule to every need. Use it when the value is computed the same way whatever the need’s type, such as a verified on every need that tests name, or a fingerprint of every need. types is not a role: a rule for some need types only is declared in those types’ own entries (Rules per need type). A rule on a core field (status, tags, title and the other core fields, and the parent_needs link type), or on a link type named after a core field (such as parts), is not available in this release: it is reported (needs.derive_invalid), and ignored.

Rules per need type¶

One value is often computed differently on different need types. Effort is written on tasks and rolled up through stories and epics: on a task the total is the task’s own effort, on a story or an epic the sum of the totals of the needs it links. That takes two fields: the one authors write, and a derived one, which copies it on a task and sums itself over the links of a story or an epic. A need type’s own entry (Fields per need type) gives the rule to the needs of that type:

[needs.types.task]
title = "Task"
prefix = "T_"

[needs.types.story]
title = "Story"
prefix = "S_"

[needs.types.epic]
title = "Epic"
prefix = "E_"

[needs.fields.effort]
schema = { type = "number" }

[needs.fields.total_effort]
schema = { type = "number" }
nullable = false

[needs.types.task.fields.total_effort]
derive = { kind = "copy", field = "effort" }

[needs.types.story.fields.total_effort]
derive = { kind = "sum", field = "total_effort", over = "links" }

[needs.types.epic.fields.total_effort]
derive = { kind = "sum", field = "total_effort", over = "links" }

.. task:: Write the parser
   :id: TASK_1
   :effort: 3

.. task:: Write the tests
   :id: TASK_2
   :effort: 5

.. task:: Write the docs
   :id: TASK_3
   :effort: 2

.. story:: Parse the input
   :id: STORY_1
   :links: TASK_1, TASK_2

.. epic:: Ship the reader
   :id: EPIC_1
   :links: STORY_1, TASK_3

Each task’s total_effort is its effort (3.0, 5.0, 2.0), STORY_1’s is 8.0, and EPIC_1’s 10.0: a rule reading its own field over a link type reads the value each linked need computes, through any depth.

Bear in mind:

One name, one shape. A field or link type means the same thing on every need type that declares it: its type (number here) and whether it is derived belong to the field, not to a need type, and so, for a link type, do how its links are drawn, whether a dead link is allowed, whether its links are copied into links and whether it is a summary (for a link type declared on need types alone, the first need type declaring it, in the order the types are given, sets them; an entry giving another value is reported (needs.type_conflict) and ignored). A need type’s own declaration of a derived field narrows its description, schema and nullable, but gives it no default.
One field cannot do both. A derived field is closed on every need type, also where it has no rule, so the value authors write on tasks is a field of its own, effort.
Cycles. Needs linking one another in a cycle cannot compute such a value, and are reported (needs.derive_cycle).
Other need types. A need of a type without a rule holds the same value on every need of the type, set out below.

The entry is one of needs_types: the type is declared there, with its title and prefix, and declaring need types replaces the default ones (see Modeling need types); an entry naming a default type with neither title nor prefix is that type, narrowed, so [needs.types.req.fields.<field>] alone gives req a rule and keeps every default type. The rule is a derive in the type’s declaration of the field, under fields, or of the link type, under links. A type inheriting from it uses the rule too, unless it declares one of its own. A need uses the rule of its type, else derive, else none.

A need of a type without a rule holds the same value on every need of the type:

beside rules for some need types	a need of a type without a rule holds
a `default`	the default, in the field’s type (`default = 0` is `0.0` in a `number` field); the field’s `schema` checks it, as it checks any default (but a link type’s `default = []`, which is its “no links”, as no default is)
no `default`, nullable (as the field is declared: a needs_fields entry is unless `nullable = false`; a needs_extra_options entry without a `schema` is not)	`None`
no `default`, `nullable = false`	the field’s empty value (`""`, `0`, `0.0`, `False` or `[]`; no links for a link type), which the `schema` does not check

The default must be a literal value, and is given only where need types alone have rules: beside derive every need type has a rule, and a default is reported as needs.derive_invalid, and ignored. A default that is a [[…]] or a variant, and predicates, are reported beside such rules too, and ignored: a value that varies per need is what a rule is for.

derive_by_type on the field’s or link type’s own entry is the spelling of the same rules for a need type the project does not declare, and an older one for a type it does: a table from need type, by its name (the value type holds, as a filter reads it: type == 'story'), to a rule, as derive takes one, such as derive_by_type.story = { kind = "sum", field = "total_effort", over = "links" }. A derive in a type’s own entry is the field’s derive_by_type entry for that type, and for each type inheriting from it that gives none of its own; its needs.derive_invalid warnings name the entry it came from. derive_by_type alone gives a need type a rule without an entry for the type: a type a GitHub service adds, which an entry would replace (a type an extension adds takes an entry of its own too, which narrows it, narrowing an extension’s type); and it alone gives a type a rule that the types inheriting from it do not get.

Rules per need type are checked when the configuration is read, one needs.derive_invalid per entry: a derive_by_type key naming no need type is ignored (a type’s title is not its name, and the warning says which is meant); an empty derive_by_type, or one that is not a table, adds no rule: the field stays derived, every need type using the derive beside it, or, without one, no rule; an entry that cannot be read, as a derive that cannot be read, leaves the needs of its type at the empty value, and the other types are unaffected. Rules reading one another’s own fields in a circle are looked for per need type, among the rules each type uses: a derive closing such a circle on some types only is reported naming them, and applies on every other type.

Each need is computed in the step of its own rule, so one field may be computed after every other on one type (after = "derived") and not on another: a value that reads the field waits for the rule of the need it reads, and only the needs whose rule comes later are reported (needs.derive_scope).

The kinds at a glance¶

Pick a kind by what the need should show:

a value written on another need, such as the owner of the specification a test links to: copy;
a total, such as the effort of a story’s tasks: sum; how many, such as the open tests of a requirement: count;
the worst or best value, such as the highest risk of a requirement’s implementations: min or max;
a verdict, such as whether every test of a requirement passed: any or all;
the values of several needs as one list, such as the requirements a test traces to: collect;
links that follow from the needs’ fields, or from the references in their text: links, content_links;
the links of several link types as one: union;
the far ends of a chain, such as the requirements a test result is evidence for, as links: copy or collect on a link type;
a value that changes when a need does, to notice a change since a review: hash or fingerprint.

kind	declared on	required roles	optional roles	result, which the field’s `schema` must hold
`copy`	a field	`field`	one of `from`, `over` and `join`; `select` and `where` (with `over` or `join`); `include_self` (with `join`)	the type of the field read; with `select = "list"`, an array of it
	a link type	`field = "id"`, `over`	`where`; `select = "list"`	a link list: a link to each need `over` names (A chain as links: copy and collect on a link type)
`sum`	a field	`field`, `over`	`where`	a `number` (or an `integer`, when the field read is one)
`count`	a field	`over`	`field`, `where`	an `integer`
`min`, `max`	a field	`field`, `over`	`where`; `transitive`, `include_self`	the type of the field read, which must be a number or a string with an `enum`
`any`, `all`	a field	`over`, and exactly one of `field` (a boolean field) and `test`	`where`, `empty`	a `boolean`
`collect`	a field	`field`, `over`	`where`	an array of the item type of the field read
	a link type	`field` (a link type), `over`	`where`	a link list: the links `field` holds on each need `over` names (A chain as links: copy and collect on a link type)
`hash`	a field	`fields`	(none)	a `string`, computed after every other rule of the need
`fingerprint`	a field	(none)	`exclude`	a `string`, computed after every other rule of the need
`links`	a link type	`where` or `join`	the other of the two; `include_self`; `include_parts` (not with `join`)	a link list
`content_links`	a link type	(none)	`from`, `where`	a link list
`union`	a link type	`fields`	(none)	a link list: the links of the link types `fields` names (A union of link types)

Every kind that computes a value, but fingerprint, also takes after = "derived" (copy and collect on a field only), which computes the field after every other rule of the need, as hash and fingerprint always are.

Note

Three things are called copy, and only the first is a kind:

the copy kind of a rule, derive = { kind = "copy", … }, which reads a field (above);
the copy dynamic function, [[copy("x")]] written in a need, whose declared form is mostly that kind (see Rules and inline calls);
the copy = true option of a needs_links entry, deprecated, which copies the link type’s links into links; its declared successor is a union, not the copy kind.

Values from linked needs¶

These kinds compute a field’s value from the needs a rule reads, its candidates: the needs over names (see How rules are computed), and for copy also the need from names, this need itself, or the needs a join matches. Each stores one value per need (What a rule costs). over follows a link type either way: over = "links" reads the needs this need links to, and over = "tests_back" the needs that link to this need in their tests, such as the tests of a requirement when the tests name the requirement.

copy¶

copy shows on a need a value written on another need, such as the owner of the specification a test links to, so that a table of tests shows whom to ask:

[needs.fields.owner]
schema = { type = "string" }

[needs.fields.spec_owner]
schema = { type = "string" }
derive = { kind = "copy", field = "owner", over = "links" }

A test whose links names a specification with owner alice holds spec_owner "alice".

copy reads one field: this need’s own (with neither over nor from), that of the need from names, or that of the needs over names or a join matches. A copy of the need’s own field is where a roll-up starts, as on the tasks of Rules per need type. On a link type, copy of id links the needs over names instead of listing their ids (A chain as links: copy and collect on a link type).

copy over several targets takes the one with the lowest need id among those that set the field (select = "first"), the same and a needs.derive_unique warning when several set it ("unique"), or every value in the order written (in need-id order over a join), a list’s items each ("list").

With over or join, a where narrows the targets first, as it narrows the candidates of the other kinds, such as the owner of the linked specifications that are not retired:

[needs.fields.active_owner]
schema = { type = "string" }
derive = { kind = "copy", field = "owner", over = "links", where = "status != 'retired'" }

select reads only the targets where holds on, so "first" is the lowest-id target that passes it and sets the field. A target it cannot be evaluated on is dropped, without a warning. A copy of this need’s own field, or of the need from names, reads one need and takes no where (needs.derive_invalid).

Added in version 9.2.0: where on copy, with over or join.

Starting a list roll-up one rung up¶

To show on every need of a chain the requirements it traces to, start the list on the rung above the requirements with copy of id and select = "list", then collect it on each rung below:

[needs.links.satisfies]
[needs.links.implements]
[needs.links.tests]

[needs.fields.req_ids]
schema = { type = "array", items = { type = "string" } }

[needs.types.spec.fields.req_ids]
derive = { kind = "copy", field = "id", over = "satisfies", select = "list" }

[needs.types.impl.fields.req_ids]
derive = { kind = "collect", field = "req_ids", over = "implements" }

[needs.types.test.fields.req_ids]
derive = { kind = "collect", field = "req_ids", over = "tests" }

A specification satisfying REQ_1 and REQ_2 holds req_ids ["REQ_1", "REQ_2"], and so do the implementations of it and their tests. The list starts on the specification, not on the requirement: there a plain copy of id reads one string, which an array field does not hold, and select takes a list only with over or join; both are refused (needs.derive_invalid).

The same chain read upwards gives each need the tests that reach it, the back links of each rung (<link type>_back) as over:

[needs.fields.reached_by]
schema = { type = "array", items = { type = "string" } }

[needs.types.impl.fields.reached_by]
derive = { kind = "collect", field = "id", over = "tests_back" }

[needs.types.spec.fields.reached_by]
derive = { kind = "collect", field = "reached_by", over = "implements_back" }

[needs.types.req.fields.reached_by]
derive = { kind = "collect", field = "reached_by", over = "satisfies_back" }

With TEST_1 and TEST_2 testing the implementation, the implementation, the specification and both requirements hold reached_by ["TEST_1", "TEST_2"].

A list of ids suits a table column or a filter ("REQ_1" in req_ids). To have the far ends as links instead, drawn on the need and with back links on the requirements, declare the downward rules on a link type, each collect reading that link type (A chain as links: copy and collect on a link type): its back links then give each requirement every need reaching it, which a link type’s own rule could not, as it reads no back links.

Matching a field: the join role¶

Added in version 9.1.0.

Some needs refer to others by a value rather than by a link: a test names the test cases it runs, and the specification describing each case holds the same name. join = { field = "<field>", equals = "<own field>" } makes the candidates of a links or copy rule the needs whose field equals one of this need’s equals values, such as the specifications a test names in its test_case field by their case_name:

[needs.fields.case_name]
schema = { type = "string" }

[needs.fields.test_case]
schema = { type = "string" }

[needs.fields.owner]
schema = { type = "string" }

[needs.links.covers]
derive = { kind = "links", join = { field = "case_name", equals = "test_case" } }

[needs.fields.owners]
schema = { type = "array", items = { type = "string" } }
derive = { kind = "copy", field = "owner", join = { field = "case_name", equals = "test_case" }, select = "list" }

.. spec:: Login
   :id: SPEC_LOGIN
   :case_name: login
   :owner: alice

.. spec:: Login, remembered
   :id: SPEC_REMEMBER
   :case_name: login, remember

.. spec:: Logout
   :id: SPEC_LOGOUT
   :case_name: logout
   :owner: bob

.. test:: Session test
   :id: TEST_SESSION
   :test_case: login, logout

TEST_SESSION covers SPEC_LOGIN, SPEC_LOGOUT and SPEC_REMEMBER (whose case_name holds login too), and its owners are ["alice", "bob"]; a spec, which has no test_case, covers nothing.

Bear in mind:

Values, not ids. A join compares values exactly, so a test naming Login matches no login; where the test can name the specification’s id, a written link is simpler.
Incremental builds. The pages of the needs holding a copy with join are re-written on every change of a need (Incremental builds); a links rule, with join too, makes it re-write far more pages (What a rule costs).

The rules:

The match. Both fields are read the same way: a string is split on , and each value stripped of whitespace, an array of strings is taken item by item (an item is not split again), and an empty value is dropped. A need matches when its field and this need’s equals share a value, compared exactly (case-sensitive).
Candidates. Each need that matches, once, however many values it shares, in need-id order. This need is left out unless include_self = true. Needs only: a need part is never a candidate (a need with parts matches as itself), and include_parts cannot be given beside join.
links and copy. A where beside join narrows the candidates: it is tested on each, and a candidate it does not hold on, or cannot be evaluated on, is dropped without a warning. links links the candidates left: it requires where or join, and takes both. copy reads the candidates left as it reads the needs a link type names: select takes the lowest-id candidate that sets the field ("first", the default), the same and a needs.derive_unique warning when several set it ("unique"), or every value ("list"); with no candidate it finds no value ([] with "list"). copy takes one of from, over and join.
The fields. field and equals each name a field holding a string (an enum included) or an array of strings: one of needs_fields, or a core field such as id, title, type, status or tags. A link type, a <link type>_back, and a number or a boolean field are reported as needs.derive_invalid.
The order. A join reads equals on this need, and field (with copy’s field, and the names of a where) on every need. A copy with join is computed after every need computing those fields in its step; a links with join is computed with the link fields, before any other field, so one whose fields a rule, a [[…]] or a variant computes is not run (needs.derive_scope, see Reads that cannot be ordered). A copy whose equals is its own field, or reads it back through other rules, can be computed on no need, and is reported when the configuration is read; one whose field is its own field reads its own result on every need, a cycle (needs.derive_cycle). The needs are indexed by field once for the link fields and once for the other fields (and once more for the rules computed last), shared by every rule and call on it, however many needs hold the rule.

The inline spelling is links_from_match: [[links_from_match("case_name", "test_case")]] in a link field of TEST_SESSION links the same needs. The nearest declared form of sphinx-test-reports’ tr_link(test_option, target_option) is join = { field = "<target_option>", equals = "<test_option>" } with include_self = true, or inline [[links_from_match("<target_option>", "<test_option>", include_self=True)]]: the arguments come in the swapped order, the field read on every need first. tr_link links a need once per value that matches it, in the order the project holds the needs, compares each need’s target_option whole (so an array, a number or a comma-separated value never matches, and an unstripped one only as written), and fails when this need’s test_option is an array or a number; links_from_match reads both sides by value and fails the call on a number or boolean field.

sum¶

sum adds up field over the candidates. Use it for a total over linked needs, such as the hours of a requirement’s open implementations:

[needs.fields.hours]
schema = { type = "number" }

[needs.fields.open_hours]
schema = { type = "number" }
derive = { kind = "sum", field = "hours", over = "links", where = "status != 'done'" }

A need linking implementations of 3 hours (done), 5 hours and 2 hours (both open) holds open_hours 7.0.

A candidate that does not set hours, or holds no number there, adds nothing, and a need with no candidate holds 0.0 (How rules are computed). To roll a total up through several rungs, sum the derived field itself on each rung (Rules per need type).

sum adds in the order of the candidates.

count¶

count counts the candidates, or with a field the candidates that set it. Use it to know on one need how many needs point at it, such as the open tests of a requirement:

[needs.links.tests]

[needs.fields.open_tests]
schema = { type = "integer" }
derive = { kind = "count", over = "tests_back", where = "status == 'open'" }

A requirement that one open and one passed test name in tests holds open_tests 1.

The tests name the requirement, so the requirement reads them through the back links, tests_back. A count stores one number per need (What a rule costs): the needs themselves are in tests_back already.

min and max¶

min and max give the least and the greatest field among the candidates. Use them for the worst or best value among linked needs, such as the highest risk of a requirement’s implementations:

[needs.fields.risk]
schema = { type = "string", enum = ["low", "medium", "high"] }

[needs.fields.max_risk]
schema = { type = "string", enum = ["low", "medium", "high"] }
derive = { kind = "max", field = "risk", over = "links" }

A need linking implementations of risk medium and low holds max_risk "medium".

min and max order numbers by value, and a string by the order of its enum: list the enum from least to greatest, as low, medium, high above. A rule reading a field with no order, such as a string without an enum, is refused (needs.derive_invalid).

With transitive = true, min and max read every need reachable through over, each once: a need on a cycle reaches itself, so every member of a cycle has the same value; include_self adds the need’s own field. A rule whose field is its own field (a roll-up of one field per need type, see Rules per need type) reads the value each reached need computes, so needs linking one another in a cycle are reported (needs.derive_cycle) rather than closed over. With include_self = true such a rule reads its own value on every need, which is a needs.derive_cycle on each: an own-field roll-up leaves include_self off. A transitive rule reads needs it does not name, so an incremental build re-writes the pages of the needs holding it on every change of a need (Incremental builds).

any and all¶

any and all say whether test holds (or a boolean field is true) on any candidate, or on every one. Use them for a verdict over linked needs, such as whether every test of a requirement passed:

[needs.links.tests]

[needs.fields.verified]
schema = { type = "boolean" }
derive = { kind = "all", test = "status == 'passed'", over = "tests_back", where = "type == 'test'" }

A requirement whose tests all passed holds verified true, one with a failed test false, and one no test names true, as all over no candidate is.

On a requirement nothing tests, that true reads as “every test passed”. empty names the value over no candidate instead, such as empty = false:

[needs.fields.verified]
schema = { type = "boolean" }
derive = { kind = "all", test = "status == 'passed'", over = "tests_back", where = "type == 'test'", empty = false }

Now the requirement no test names holds verified false, and so does one that only a specification names, which where drops: no candidate is what is left once where has dropped the needs it does not hold on. Over candidates empty is not read, and a candidate that leaves a boolean field unset still reads as false. Without empty, each kind keeps its own value over no candidate.

Added in version 9.2.0: The empty role.

collect¶

collect gathers the values of field over the candidates into one list. Use it to show on one need what its linked needs hold, such as the tags of a requirement’s implementations:

[needs.fields.linked_tags]
schema = { type = "array", items = { type = "string" } }
derive = { kind = "collect", field = "tags", over = "links" }

A need linking implementations tagged io, parser and io, cache holds linked_tags ["io", "parser", "cache"].

collect keeps each value once, in the order of the candidates, a list’s items each. Collecting a derived list field over the next rung of a chain carries the list down it (Starting a list roll-up one rung up). On a link type, collect of a link type carries links down the chain instead (A chain as links: copy and collect on a link type).

Link lists¶

These kinds compute a link type’s links, and are declared on a link type, in its needs_links entry. Their links are links like written ones: shown on the need, read by filters, tables and diagrams (a summary link type only where named), and answered by back links on the needs they point to. Each computes with the link fields, so its links are in the back links (<link type>_back) as written ones are.

Bear in mind:

Field form or link form. A list of ids in a field suits a table column or a filter; links add what links have, the back links above all. Where nobody reads the back links, a field is lighter.
Cost. A link type whose rule links many needs to many stores a list on each need, and a back link on each target; needs.derive_dense reports a link type that grows past needs_derive_max_links_per_need links per need of the project (What a rule costs).
Incremental builds. Some of these rules make an incremental build re-write far more pages on each change of a need; What a rule costs says which.

Summary link types¶

Added in version 9.2.0.

Some links are shortcuts across the trace rather than steps in it: a test result linked straight to the requirements it is evidence for, three links up (test result, test, specification, requirement), or a risk linked to what it is about. They are what you want in a table column or a filter (“the requirements this result covers”), but drawn beside the links they shortcut, they would show every path twice.

Mark such a link type as a summary in its needs_links entry:

[needs.links.covers]
summary = true

A link type declared on need types alone is marked in the entry of the first need type declaring it:

[needs.types.result.links.covers]
summary = true

Wherever you name it, a summary link type works as any link type: a filter ("REQ_1" in covers), a needtable column, a rule’s over or field, a needflow’s :link_types:, a need type’s count of the links it receives (Links a type must receive, which counts every link, computed ones included); and a need holds its links and back links as usual. What changes is the views that take every link type without being told which: a needflow without :link_types: leaves it out, and the needs.json schema marks it "summary": true so that other tools can do the same.

You rarely write such links by hand: a copy or collect on the link type computes them from the chain (A chain as links: copy and collect on a link type), and a link type computed that way is a summary whether or not its entry says so. A link type written in the needs may be one too, such as a mitigates that says what a risk is about.

summary belongs to the link type, so it holds on every need type. On a link type declared on need types alone, the other need types declaring it may repeat it or leave it out; one giving another value is reported (needs.type_conflict) and its entry ignored, as for the rest of the link type’s shape. On a link type declared in needs_links, summary in a need type’s own links entry is reported (needs.config) and ignored. A union cannot name a summary link type as a member, and summary is no key of a rule (each needs.derive_invalid).

links¶

links links every need a where holds on, or the needs a join matches (Matching a field: the join role). Use it for links that follow from the needs’ fields rather than being written, such as a link to every open requirement; keep its where selective (What a rule costs):

[needs.links.open_reqs]
derive = { kind = "links", where = "type == 'req' and status == 'open'" }

With one open requirement, REQ_1, every other need holds REQ_1 in open_reqs, REQ_1 holds no link there (a need is left out of its own), and its open_reqs_back lists every other need.

Bear in mind:

It is stored on every need holding the rule. With a where and no join, every such need links to every need the where passes (but itself, unless include_self); to show that set in one place, a needtable or needlist with the same filter shows it once. Declare the rule only for the need types that use it (Rules per need type).
Incremental builds. A links rule makes an incremental build re-write far more pages (What a rule costs).

Its candidates are every need, and with include_parts every need’s parts (after it, in part-id order), in need-id order; this need and its parts are left out unless include_self. With join, the needs that match (Matching a field: the join role), in the same order. needs.json writes the list sorted naturally, as every link list.

content_links¶

content_links links the needs a need’s content references with need. Use it so that the references written in a need’s text are links too, with back links: a specification whose text cites the requirement it refines then shows on that requirement, without a second, written link to keep in step with the text.

[needs.links.mentions]
derive = { kind = "content_links" }

A need whose content says :need:`REQ_1` holds REQ_1 in mentions, and REQ_1 holds it in mentions_back.

Its links are the :need: references of this need’s content (or of the content of the need from names), in document order, each once: a reference to a part is a part link, a reference to this need is kept, a reference to no need is kept (and reported as any dead link), a disabled one (!ID) and those inside a nested need are not. With where, only the references to needs it holds on are kept (a part through its need). A content_links rule with from makes an incremental build re-write far more pages (What a rule costs).

A union of link types¶

Added in version 9.1.0.

A trace is often spread over several link types: a requirement is satisfied by specifications, implemented by code and tested by tests, and one view should read them all at once: a needflow, a filter or a needtable column (see Reading it, below). A link type deriving union holds the links of other link types, its members, which fields names:

[needs.links.blocks]

[needs.links.tests]

[needs.links.all_links]
outgoing = "all links"
incoming = "all links from"
derive = { kind = "union", fields = ["links", "blocks", "tests"] }

A need whose links is REQ_1, whose blocks is REQ_2 and whose tests is REQ_1 then holds REQ_1, REQ_2 in all_links, and REQ_1 and REQ_2 hold it in all_links_back. It is the declared spelling of “every outgoing link of these types”, which copy = true gave by copying into links (deprecated, see needs_links).

Bear in mind:

The members keep their own links. A union adds a link type; links and each member are unchanged, so a view that names a member still draws it with its own style.
It is not counted twice. The links it holds again are not counted as dense (below), and where its rule computed it, its dead links are reported once, at the member.
Members only. A union reads link types: not a field, not back links, not parent_needs, and not a summary link type.

The rules:

Members. Any link type the need has: links, one of needs_links or needs_extra_links, one need types declare alone, a derived link type, another union – each computed before the union, whether its links are written, computed by a rule, a built-in function such as links_from_filter, a variant, or added by a needextend. A member filled by one of your own functions (such as sphinx-test-reports’ tr_link), or by a call whose filter cannot be read, such as a links_from_filter whose filter names needs, is computed after the other built-in functions of its step, with your own functions, in need-id order (see Processing order), and the union over it right after the last such member – a union of that union after that. Those computed after the union read it complete; those computed before it read it empty: such a call whose filter names the union (by name, or as its own need’s current_need key) is not run, and is reported (needs.derive_scope), as is a copy reading it, when it runs, and so are your own functions, a read through needs, and one of current_need by a key that is not written out, as they run (Reads that cannot be ordered). A built-in function or rule of the link fields’ step that reads such a union cannot wait for it: it is not run, and is reported (for a links rule, on every need computing it). parent_needs and the back links (<link type>_back, built after the link types) are not members, nor is a field, nor a summary link type (summary in needs_links, Summary link types), whose links are shortcuts across the trace, not steps in it, and a member is named once; the union itself, or two unions naming each other, can be computed on no need. Each is one needs.derive_invalid at load, and the union holds no links. On a need of a type that does not have a member, the member holds no links, and adds none.
The links. Each member’s links, in the order fields names the members, each member’s in its own order; a link is kept the first time it comes, a link being its target, its part and its condition: REQ_1 from two members is one link, while REQ_1, REQ_1.p1 and REQ_1[status == 'open'] are three. Parts and conditions are carried as written. needs.json writes the list sorted, as every link list; the order above is what a rule or a [[…]] reading the union in the pass sees.
Dead links and conditions are the members’. Where its rule computed it, a union reports no dead link and no failed link condition of its own: each is reported once, at the member, which decides with its own allow_dead_links whether a dead link is allowed. So a dead link of a member allowing dead links is kept in the union and makes no dead link forbidden (has_forbidden_dead_links; has_dead_links counts it, as every dead link), it is drawn in a need’s layout as the member allows it, and an allow_dead_links on a link type all of whose rules are unions is refused (needs.derive_invalid) and ignored; beside a rule of another kind, for another need type, it is that rule’s, and beside a union that cannot be read (a member refused, a cycle), which is no union and holds no links, it is kept. A union cycling on some need types only is still a union on the others, so the key beside it is refused, which the cycle’s finding says. target_types is refused on it, as on every derived link type: the members’ own apply.
When. In the link fields’ step, after every needextend: a link a needextend adds to a member is in the union. The union itself is closed, as every derived field: links written in it, or by a needextend, are reported (needs.derive_authored) and ignored.
Imported needs keep their data. No rule runs on a need from needimport or needs_external_needs: it holds the union its needs.json carries, else no links, whatever its members hold. Those links are not its members’, so they are checked as any link type’s, under the union’s own allow_dead_links: a dead one is reported, under the union, and is a forbidden one.
Not counted as dense. needs.derive_dense (What a rule costs) counts the links its members compute, not the union’s list, which holds them again.
It is left out of every fingerprint, as every derived link type is, and it may be declared per need type (a need type’s links entry, Fields per need type, or derive_by_type).

Reading it. A union is read where links holding the copies was read, by its name:

needflow (and needsequence, needgantt’s starts_after_links): :link_types: all_links. A needflow drawing every link type (or the members and the union) draws an edge of the union beside each member’s, so name either the union, which draws one style, or its members, which keep theirs.
Filter strings: "REQ_1" in all_links; a rule’s over = "all_links" or over = "all_links_back".
needtable columns: all_links and all_links_back (OUTGOING and INCOMING stay links).
A need’s layout shows it as one more row, as every link type with links; meta_links_all(exclude=[...]) leaves it out.

Some readers read only links in this release, and take no link type yet: check_linked_values, calc_sum with links_only, and the need_outgoing and need_incoming roles written in a need’s content. While copy = true is in use they see the copies, as before. A project that relies on them keeps copy = true until 10.0, and with it the needs.deprecated warning; suppress_warnings = ["needs.deprecated"] silences it, and every other deprecation with it.

A chain as links: copy and collect on a link type¶

Added in version 9.2.0.

A test result is evidence for the requirements its test traces to, several links up: test, implementation, specification, requirement. A shortcut across the trace gives each need the far ends of its chain as ordinary links, shown on the need, and on each requirement as back links: “the needs that reach me”. copy and collect declared on a link type carry a chain’s far ends down it as links, rung by rung, where on a field they carry them as an array of ids (Starting a list roll-up one rung up):

[needs.links.satisfies]
[needs.links.implements]
[needs.links.tests]

[needs.links.chain]
outgoing = "traces to"
incoming = "is reached by"

[needs.types.spec.links.chain]
derive = { kind = "copy", field = "id", over = "satisfies" }

[needs.types.impl.links.chain]
derive = { kind = "collect", field = "chain", over = "implements" }

[needs.types.test.links.chain]
derive = { kind = "collect", field = "chain", over = "tests" }

A specification satisfying REQ_1 and REQ_2 links both in chain, and so do the implementations of it and their tests; REQ_1 holds every one of them in chain_back, “the needs that reach me”, which the field form gives only as a second roll-up up the chain’s back links. The field form stays the one for a table column or a filter of ids; the link form adds what links have: the links rendered on the card, back links, and in needs.json a link list with its _back.

Bear in mind:

It is a summary. Its links skip the steps between, so a needflow draws them only when :link_types: names the link type, and a union cannot read it (below).
Cost. Each need stores a link per far end its chain reaches, and each far end a back link from every need reaching it; on a deep, wide trace, the field form stores the same ids without the back links.

The rules:

copy. field = "id" and over: a link to each need over names, in the order written, each once; a link to a part links the part’s need, and a link to no need is skipped. select is "list" (a link list holds every target), which it need not say.
collect. field, a link type (a derived one too, which carries a chain rung by rung), and over: the links field holds on each need over names, in their order, each once by its target and its part. A link’s condition is dropped: it qualifies the link it is written on, not the chain. A dead link is carried, and reported where it is written, as that link type allows dead links, not again here: it sets has_dead_links (and has_forbidden_dead_links) on the need writing it only, and a need’s layout draws it forbidden on every rung exactly where the link type it is written in does not allow it. allow_dead_links beside such a rule is refused (needs.derive_invalid) and ignored, as beside a union.
where tests each need over names, as on the other kinds.
When. With the link fields (Processing order), so each reads what is known then: over names no back links (built after the link types), and field no value field and no back links. Each is refused at load (needs.derive_invalid), and the link type holds no links. A need is computed after the needs it reads; needs reading one another’s chain around a ring, or a need naming itself, are each reported (needs.derive_cycle) and hold no links, as for a roll-up of a field’s own value, and a need reading one of them reads its empty list, unreported. A where naming a value field a rule computes, or back links, cannot wait for them: each need whose over names a need is reported (needs.derive_scope) and holds no links.
Over a derived link type. A union is read through its members, so a chain may climb several link types at once, over = "trace_up" with trace_up the union of satisfies, implements and verifies, and a collect of its own link type over it is computed rung by rung, a ring through it reported as above. Over a link type a links, content_links, copy or collect rule computes (or a union with such a member), or where a call computes a need’s over with the link fields ([[links_from_filter(...)]]), which needs it names is known only once it is computed, so the rule reads field on every need computing it: a copy, or a collect of another link type, is computed; a collect of its own link type would read itself on every need, and is refused at load (needs.derive_invalid) over such a rule, or reported as a cycle where a call computes over.
A summary. The link type is a summary link type (Summary link types), whether its entry says so or not: a needflow draws it only when :link_types: names it, and a union cannot name it. summary = false beside such a rule is refused (needs.derive_invalid) and ignored: the rule applies.
Cost. A leaf stores a link per need its chain reaches, depth times fan-in, and each of those a back link: needs.derive_dense counts them (What a rule costs). It is left out of every fingerprint, as every derived link type is. A collect, or a copy over a link type another rule computes, links needs its need does not name, so once a need of the project holds one, an incremental build re-writes every page holding a need, a view or a need role on each change of a need (Incremental builds).

Digests¶

A review holds only while what was reviewed stays the same: a specification checked against a requirement should be checked again once the requirement changes. A digest is a value that changes when the need does, or, for a hash, when one of the fields it lists does, so a link condition naming the value seen at the review reports the need once it changes (Detecting a changed need). These kinds compute a string from the need’s own values, after every other rule of the need.

Bear in mind:

Which values count. A hash reads only the fields it lists; a fingerprint reads the whole need, so a change of any value it includes moves it, a typo fixed in the content too.
A fingerprint leaves derived values out. A fingerprint never includes a derived field or link type, so a rule’s result changing on a linked need does not move it.
Cost. Each stores one string per need, computed from that need’s own values.

hash¶

hash digests the fields fields lists, in that order. Use it where only some of a need’s fields matter, such as a review that a link condition pins to a requirement’s title and status: unlike a fingerprint, it does not change when another field does.

[needs.fields.title_status]
schema = { type = "string" }
derive = { kind = "hash", fields = ["title", "status"] }

A need titled Größe with status open holds 354a4be93cba05e4f1332b19868b4397b200f66d53495ae40f7502e5e638a740, and a link condition naming that value holds until the need’s title or status changes.

Its value is the lowercase hex SHA-256 of the UTF-8 text of one JSON array: the listed fields’ values in the listed order, each as needs.json holds it (an unset field null), with no whitespace, non-ASCII characters not escaped, and numbers as Python writes them (3.0, 1e+16). A link list is written as the ids it links to, without their conditions, sorted and without duplicates, so a need linked with and without a condition is hashed once. For fields = ["title", "status"] on a need titled Größe with status open, the text is ["Größe","open"].

fingerprint¶

fingerprint digests the whole need, so it changes whenever the need does. Use it to notice that a need changed since it was reviewed (Detecting a changed need).

Its value is the lowercase hex SHA-256 of the UTF-8 text of one JSON object holding the whole need, its keys sorted, written as for hash (no whitespace, non-ASCII characters not escaped, numbers as Python writes them, and a number field’s value as a float).

It includes id, type, title, status, tags and content (as needs.json holds it: a need nested in another’s body is part of the parent’s content, so editing or moving the nested need changes its parent’s fingerprint too), every field the project declares in needs_fields or needs_extra_options, and every link type, links among them; none of them derived.
It never includes a derived field or link type, one with a rule on any need type (a derive, also in a need type’s entry, or a derive_by_type; so no hash and no other fingerprint), a back link list (<link type>_back), parent_needs, parts (a part’s text is in content), constraints, where the need is (docname, lineno, sections), how it is shown (layout, style, collapse, hide, the templates), what sphinx-needs records about it (is_external, has_dead_links and the like), or a field an extension or a service registers, such as needgantt’s duration and completion, unless the project declares it itself in needs_fields or needs_extra_options.
A name enters only where the need’s value differs from the field’s default, compared as JSON text: the field’s default, else null for a nullable field (a default that is a [[…]] or a variant is no literal default, and so counts as none), and for a field that is neither, nothing: it always enters. status and tags are compared against theirs (null and [] unless configured), content against "", and id, type and title always enter. A link type, and tags, without a literal default are compared against []. A number field’s value and default are compared, and written, as floats: default = 1 is 1.0. So a value written equal to its default is the same as leaving it unset, while "", false, 0 or [] differ from a null default, and -0.0 from 0.0.
Declaring a new field or link type moves no fingerprint while the needs leave it unset, if its default is a literal or there is none. A default that is a [[…]] or a variant, and predicates, give the needs a value, which enters as a written one would: adding owner with default = "[[copy('title')]]" moves the fingerprint of every need, and adding severity with predicates = [["type == 'spec'", "major"]] that of every spec.
Changing a field’s default moves the fingerprints of the needs that wrote the old or the new default out, and not of those that rely on it, although their value changes.
A link list is written as for hash, and compared with its default the same way: the ids it links to, without their conditions, sorted and without duplicates. So a condition pinning another need’s fingerprint never moves the fingerprint of the need holding it, while linking another need does.
exclude leaves names out, such as exclude = ["id"], under which two needs with the same contents have the same fingerprint.
It is computed after every other rule, from the need’s final values (after every needextend, variant and [[…]]), and reads no derived field, so two fingerprints with the same exclude hold the same value.
No later release changes which names a fingerprint includes, when a name enters, or how the text is written: a fingerprint written down stays current until the need, or the configuration of the fields it reads, changes.

For a need REQ_1 of type req, titled Größe, with status open and no other value set, the text is {"id":"REQ_1","status":"open","title":"Größe","type":"req"}.

Detecting a changed need¶

A fingerprint changes whenever its need does, so a link condition naming the fingerprint a need had when it was reviewed reports the need once it changes:

[needs.fields.fp]
schema = { type = "string" }
derive = { kind = "fingerprint" }

.. spec:: Reviewed against REQ_1
   :id: SPEC_1
   :links: REQ_1[fp == "3f9a1c2b7d4e0a8c6f1e5d2b9a7c4e3f8d6b1a0c9e7f5d3b2a4c6e8f0a1b3c5d"]

While REQ_1 is unchanged, the condition holds. Once its title, content, status or any other value it includes changes, its fingerprint does, and the build reports needs.link_condition_failed at SPEC_1; writing the new fingerprint into the condition accepts the change. Writing only the start of the fingerprint is enough to tell a change apart: REQ_1[fp.startswith('3f9a1c2b7d4e')].

The roles¶

Each operand of a rule goes under a role name, the same for every kind that takes it: over always names the links to follow, where always narrows the candidates. The table is the reference; each kind’s section above shows its roles in use.

role	value	meaning
`field`	a field name	The field read on each candidate: a field (one of needs_fields, or a core field such as `title`), a link type, or `<link type>_back`. On `copy` with neither `over` nor `from`, this need’s own field.
`fields`	a list of field names	`hash`: the need’s own fields hashed, in the listed order (fields and link types; not a `<link type>_back`). `union`: the need’s own link types whose links it holds, in the listed order, each named once (derived ones and other unions included; not `parent_needs` or a `<link type>_back`).
`exclude`	a list of field names	`fingerprint`: the names it leaves out, each a name it includes, once; an empty list leaves nothing out.
`over`	a link type	The candidates: the needs this need’s `<link type>` names, in the order written, a need named twice counted twice; or, for `<link type>_back`, the needs that link to this one, in need-id order, each once. A link to a need part reads the part’s need; a link to no need is skipped.
`from`	a need id	The one need whose `field` (`copy`) or content (`content_links`) is read.
`join`	`{ field = …, equals = … }`	`links`, `copy`: the candidates are the needs whose `field` equals one of this need’s `equals` values, each once, in need-id order. Both are text fields, a string split on `,` or an array of strings, compared value by value (see Matching a field: the join role).
`where`	a filter string	Narrows the candidates: each is kept when the filter string holds on it, before `copy`’s `select` reads them. On `links`, the filter every need is tested with; on `copy`, only with `over` or `join`.
`test`	a filter string	`any` / `all`: what is asserted of each remaining candidate. Over no candidates, `all` is `true` and `any` is `false` (see `empty`).
`empty`	`true` / `false`	`any` / `all`: the value over no candidate, once `where` has dropped the candidates it does not hold on. Unset, `all` is `true` and `any` is `false`.
`select`	`"first"` (default), `"unique"`, `"list"`	`copy` over several targets (with `over` or `join`): the target with the lowest need id among those that pass `where` and set the field; the same, reported when several set it; or every value, as a list (the field must be an array).
`transitive`	`true` / `false` (default)	`min` / `max`: read every need reachable through `over`, each once, not only the needs one link away.
`include_self`	`true` / `false` (default)	`min` / `max` with `transitive = true`: count this need’s own `field` too. `links`: this need (and its parts) may be linked. `copy` with `join`: this need may be a candidate.
`include_parts`	`true` / `false` (default)	`links` without `join`: the parts of every need are candidates too, a passing part linked as `<need>.<part>`.
`after`	`"derived"`	Compute the field after every other rule of the need.

A where or test reads the candidate only: a filter naming needs, current_need or a c. check (such as c.this_doc()) cannot be declared, so that a rule means the same in every tool that reads the configuration. c.is_a('req') with a string literal is the exception: it reads the candidate’s type alone, against the project’s need types (see Filtering for a type and the types inheriting from it). A where may also read the candidate’s c.source facets (Filtering by where a need comes from (c.source)), such as where = "c.source.path is not None and c.source.path.startswith('../src/')": they read where it comes from, which no rule computes. A where that fails on a candidate excludes it without a report, so guard a facet that can be None, as that example does.

A join matches the needs whose field equals one of this need’s values (Matching a field: the join role).

How rules are computed¶

When a rule gives a value you did not expect, or a finding names a step, the order below and what a rule does with an unset value or with no candidate explain it. A rule is computed with the [[…]] of its step (see Processing order): links, content_links, union, and copy and collect on a link type, with the link fields, hash, fingerprint and the rules with after = "derived" after every other field, every other kind with the other fields; each after every value it reads (a union after the derived link types it names).

Candidates. over = "<t>": the needs this need’s <t> names, once the link fields are computed, in the order written; a need named twice is a candidate twice; a link to a need part is its need, and a link to no need is skipped. over = "<t>_back": the needs that link to this need (or to one of its parts), in need-id order, each once, as a back link list is read. where drops the candidates it does not hold on, then test is tested on the others (and copy’s select reads them). A where or test that cannot be evaluated on a candidate (a field the candidate lacks, as hours > 1 on a need without hours) does not hold on it, without a warning.
Unset values. A field is unset when it is null: an empty value ("", [], a link type with no links) is set, so count with a field counts it, and copy and collect take it. A candidate whose field is unset is skipped by sum, min, max, collect, and count with a field (which so counts the candidates that set it); any and all read it as false. sum skips a value that is not a number.
No candidate. When none is left once where has dropped those it does not hold on, sum and count are 0 (0.0 in a number field), any false and all true (or their empty), collect and copy with select = "list" [] (a list, also when no candidate sets the field); min, max and copy with select = "first" or "unique" find no value, over no candidate or over candidates none of which sets the field, and leave the field at its empty value. The values a rule computes over no candidate (and an empty link list) are checked against the field’s schema as any result is, so a count of 0 under minimum = 1 is refused, while the empty value of a rule that found no value is not checked.
Values. A result takes the type of the field holding it, as an authored value does, and as the result of a [[…]] or a variant does: an integer in an integer field, a float in a number field (a count, or the sum of an integer field, is 13.0 there), in needs.json and in a hash.

What a rule costs¶

Most rules store one value per need and cost little; a rule computing links can store far more. A rule is computed once per step for every need holding it (a links rule’s where is tested once, on each candidate alone), so writing one costs little: what costs is what it stores. Every need holding a links rule with a where (and no join) is linked to the same needs (but itself), and each of them gets a back link from every one (a join links each need to its own matches): on every need of a project of 2 000, a rule whose where half the needs pass stores about 2 000 000 links and as many back links, and needs.json grows from about 2.5 MB to about 58 MB; at twice the needs it stores four times the links. A rule that ten needs pass stores about ten links per need.

Keep where selective, or declare the rule only for the need types that use it (Rules per need type).
To show a set of needs, use a needtable or needlist with that filter: it shows the set once, where a links rule copies it into every need.
To know on one need how many needs (or which) point at it, link the other way and read the back links: { kind = "count", over = "tests_back", where = "status == 'open'" } stores one number.
links_from_filter written in every need stores what the links rule stores, and also tests its filter on every need once per call, so on many needs prefer the declared rule.
On an incremental build, once a need of the project holds a links rule (with or without join), a content_links rule with from, a collect on a link type, or a copy on a link type over a link type a rule computes, every change of a need re-writes every page holding a need, a view or a need role (Incremental builds).

A link type whose computed links number more than needs_derive_max_links_per_need (100) per need of the project is reported once, as the warning needs.derive_dense, naming the links, the needs holding the rule and the longest list.

What a derived field holds¶

This section answers what happens around a rule: when an author writes the field anyway, when a need comes from another project, and what is reported when a rule cannot be read or cannot be computed. A derived field (one with a rule: a derive, also in a need type’s entry, or a derive_by_type) is closed to authors, on every need type, also on a type it has no rule for: a value written for it in a need (a literal, or a [[…]], <<…>> or <{…}>), or set by a needextend, is ignored and reported as needs.derive_authored, at the need or the needextend. It takes no default (but the value of the need types without a rule, see Rules per need type) and no predicates. A need from outside the project’s sources, created by needimport or loaded from needs_external_needs, keeps the values it carries, as data: no rule runs on it. One lacking a derived field holds what a need of its type holds when no rule has computed it: the value of a type without a rule, else the empty value. In the schema of the needs.json, a derived field or link type is marked "readOnly": true, so a project importing the file knows the value is computed, not authored; the schema’s default of a field, or of a link type derived by type alone, is the value of a need type without a rule.

A rule is checked when the configuration is read. A rule that cannot be read is reported as needs.derive_invalid, once per rule, naming the kind and the role: an unknown kind, an unknown role, a role the kind does not take or requires, a role holding the wrong type of value, a kind computing a link list declared on a field, or one computing a value declared on a link type, a copy or collect on a link type that reads what the link types cannot wait for (an over naming back links, a field naming a value field or back links, a collect of its own link type over a link type a rule other than a union computes, a copy of a field other than id, after) or a select other than "list" on one, an over naming no link type, a field naming nothing a need has, a result the field’s schema cannot hold, a min or max reading a field that has no order, a hash naming a <link type>_back, a fingerprint whose exclude is not a list of names, names one twice, or names what a fingerprint does not include (nothing a need has, or a name it never includes, such as docname), a where or test that does not parse or reads more than the candidate, an empty where on links (it would link every need), a where or select on a copy with neither over nor join, a join that is not a table of a field and an equals naming text fields, or is given beside include_parts, a union naming what is not a link type (a field, a <link type>_back, a name declared nowhere), parent_needs, a summary link type, no link type, or one twice, summary in a rule, where it is no role, and rules reading one another’s own fields in a circle (copy without over or from, hash, and union, a union naming itself or two naming each other), which no need could compute. The field stays derived: it cannot be set, and it holds its empty value (None, or for a field that is not nullable "", 0, 0.0, False or []; no links for a link type). The empty value a derived field holds is its “no value”, as None is for a nullable field: the field’s schema does not check it, so an enum, a minimum or a pattern that refuses it is no violation. A default or predicates beside derive is reported as needs.derive_invalid too, and ignored; the rule applies. So is an allow_dead_links beside a union, whose dead links are its members’ (beside a union that cannot be read, which is no union, it is kept). So is a link type declared with copy = true while links is derived: nothing is copied into links, which holds what its rule computes; and copy = true on a derived link type, which is not available in this release: nothing is copied. A derive that cannot be read is reported for that alone, not for a key beside it; one closing a cycle on some need types only still applies on the others, so the keys beside it are still refused, which its finding says.

A rule that cannot be computed on a need leaves the field at its empty value, and is reported: a cycle of rules or calls reading one another (needs.derive_cycle, see Cycles), a read of a value its step cannot wait for (needs.derive_scope, see Reads that cannot be ordered), or a from naming no need, and a result the field’s schema refuses, such as a value outside its enum or under its minimum (needs.dynamic_function); a result is checked even when it equals the empty value, such as a count of 0 under minimum = 1. To silence a warning, add its type to Sphinx’s suppress_warnings:

suppress_warnings = ["needs.derive_invalid", "needs.derive_authored", "needs.derive_unique"]

Rules and inline calls¶

Added in version 9.0.0.

A built-in call computes a value for the one need it is written in. A derived field declares the same computation once, in needs_fields or needs_links, for every need of the project. Declare a rule when every need, or every need of a type, computes the value the same way, and the authors should not set it; write a call when one need alone computes it, or when the computation has no declared form (in the table below):

[needs.fields.total_hours]
schema = { type = "number" }
derive = { kind = "sum", field = "hours", over = "links" }

is the declared form of :total_hours: [[calc_sum("hours", links_only=True)]] written in every need, and both are computed in the same step (see Processing order): a declared rule is the field’s value, as a call written in it would be, but an author cannot set the field. The built-in functions are the per-need spellings of the kinds:

inline call	the kind it spells
`copy("x")`, `copy("x", "ID")`	`copy` with `field = "x"`, and `from = "ID"`
`copy("x", filter=…)`	no declared form: the lowest-id need the filter keeps, read from every need
`calc_sum("x", links_only=True)`	`sum` with `field = "x"` and `over = "links"`; its `filter` is the `where`
`calc_sum("x")`, `calc_sum("x", filter=…)`	no declared form: a sum over every need
`check_linked_values(result, "x", values, filter, one_hit)`	`all` over `links` (`any` with `one_hit`), with the `test` `x in values` and the `where` `filter`, giving `result`; with `one_hit`, `result` even when no linked need matches, which `any` does not
`links_from_content()`, `links_from_content("ID", filter=…)`	`content_links`, with `from = "ID"` and the `where` `filter`
`links_from_filter(filter, include_self, include_parts)`	`links`, with the `where` `filter` and the same `include_self` and `include_parts`
`links_from_match(field, equals, include_self, filter)`	`links` with `join = { field = field, equals = equals }`, the same `include_self`, and the `where` `filter`

A where that cannot be evaluated on a candidate (one lacking a field it names) drops that candidate, without a warning, in every kind and in links_from_filter; the filter of calc_sum, check_linked_values and links_from_content keeps its own behaviour.

Written in many needs, links_from_filter, links_from_content and links_from_match store a list on each, counted as a rule’s lists are: see What a rule costs.

The join role spells a correlated lookup (Matching a field: the join role, which also gives the nearest declared form of sphinx-test-reports’ tr_link), and links_from_match is its inline spelling. A copy("x", filter=…) whose filter compares a need with current_need (name == current_need['ref']) is such a lookup too, its filter tested on every need for each need it is in: a copy with join is the declared form of such a lookup (it splits the values, and finds no value rather than this need’s own when nothing matches). A rule for some need types is declared in those types’ own entries (Rules per need type), not with a role.

The conformance corpus¶

The executable half of Derived fields is a conformance corpus, packages/sphinx-needs/tests/fixtures/derive_conformance.toml in the sphinx-needs repository: one project per rule, with the values its needs hold and the warnings it gives, which sphinx-needs’ test suite runs and ubCode vendors, and is held to.