Assessment criteria

This map aims to help identify gaps in the availability of standards in the area of food and agriculture.

Users of standards (data managers, researchers, data users, developers) need above all standards that are:

  • fit for purpose (compatible with other standards, scientifically sound, complete)
     
  • endorsed, adopted and authoritative (therefore, a lack of standards in a domain is obviously a big gap, but also having a plethora of overlapping standards covering the same domain or even the same entities is a problem to be solved);
     
  • usable, including end-user-usable (available in various forms, including widgets; integrated in tools; managed on a collaborative platform) and machine-usable, i.e. open and interoperable;

In order to assess existing standards against the three categories above, we examined the assessment criteria used in two initiatives:

We selected those criteria that fit under the categories of gaps listed above (fitness, adoption, usability / openness).

1. Fitness for purpose  

Criteria:

1) Compatible
Is the standard largely compatible with related (not alternative) formal specifications in the same area of application? (from UK Gov Open Standards Board)
Does it NOT conflict with other standards describing the same entity(ies)?
Does it NOT overlap with (and not coherently map to) other existing standards in the same area?

2) Authoritative
Is this the most authoritative vocabulary in its domain or for the type of data covered?

3) Complete
Does the standard have sufficient detail, consistency and completeness to be used in datasets typical of the domain? (from UK Gov Open Standards Board)

2. Adoption / reliability

Criteria:

4) Known
Is the relevant community aware of this standard?

5) Discoverable
Is the standard easily discoverable?
E.g. is it on the VEST map of standards, can you find it if you search for “X standard”? can you find it on Google with a similar search?
(see “Discoverable (linked to from other web pages)” from ODI certificates)

6) Used in sotware
Is it used in software? In many / few?
(see “Has the formal specification been used for different implementations by different vendors/suppliers?” from UK Gov Open Standards Board)

7) Used in datasets
Is it used in datasets? I many / few? From how many different providers?

8) Endorsed
Does the standard have a strong support from different interest groups?
(from data.gov.uk assessment)

9) Regulatory
Is the standard published by a recognized standardization body or as a government directive?
(see “Is this an existing European standard or an identified technical specification in Europe?” from UK Gov Open Standards Board)

10) Long-term, sustainable
Is the maintaining organization a long-standing and authoritative body?
(see “Does the maintenance organisation for the formal specification have sufficient finances and resources to be sure of freedom from short- to medium-term threats?” from UK Gov Open Standards Board)
Is the maintainer committed to sustain and preserve the standard?
(see “Does the formal specification have a defined maintenance organisation?” from UK Gov Open Standards Board)

11) Participatory, collaborative
Is participation in the creation process of the formal specification open to all relevant stakeholders (e.g. organisations, companies or individuals)? (from UK Gov Open Standards Board)
(see also “Does the formal specification have a defined maintenance and support process?” from UK Gov Open Standards Board)

12) Maintained
Is the standard reliably maintained, indicated through its last update time?
(Does the standard use permanent URIs (or permanent URIs platforms)?)

3. Usability / openness

Criteria:

13) Available on the web
Is the standard available on the web? (from ODI certificates)
(see also “Single consistent URL for downloading data”? from ODI certificates)

14) Versatile
Is the standard available in different formats for different technologies? (e.g. XML, JSON, RDF)? (from UK Gov Open Standards Board)

15) Served by APIs
Are there APIs and web services that allow applications to:

  • Lookup terms / concepts using several parameters
  • Perform cross-walks between vocabularies
  • Extract / lookup subsets of vocabularies
  • Automatically “tag” with the vocabulary (term extraction plus advanced NLP in the case of text, other types of reasoning…)
  • Get more user- or web- friendly results (json, widgets…)

16) Manageable
Is the standard maintained on a vocabulary management platform?

17) Testable
Are there existing or planned mechanisms to assess conformity of the implementations of the formal specification (e.g. conformity tests, certifications, plugfests etc)? (from UK Gov Open Standards Board)

18) Documented
Is there official technical documentation on the standard? (see “Data is documented” from ODI certificates); Are there tutorials?

19) Supported
is the standard supported? Is there a support community?
(see “Contact details for people to provide feedback and ask questions” and “Forum or mailing list for users” from ODI certificates)

3.1 Openness

20) Machine-readable

Is the standard available in machine-readable formats?
(see “Data uses a machine readable format” from ODI certificates)

21) Meaningful
Is it serialized in appropriate vocabulary format / semantics? (OWL, SKOS, OBO...)
(see “Data published in content appropriate formats” from ODI certificates)

22) Referenceable
Does it use URIs dereferenceable as URLs as identifiers of classes, properties and instances?

23) Linked 
Is it available as Linked Data?
(see “Data uses open standard machine readable formats” from ODI certificates)
(see also “URLs used as identifiers within data” from ODI certificates)

24) Annotated
Is it accompanied by machine readable metadata?
(see “Machine readable metadata (documentation)” from ODI certificates)

25) Clearly licensed
Is the standard clearly licensed?
Is the documentation of the IPR for formal specifications publicly available (is there a clear and complete set of licence terms)? (from data.gov.uk assessment)
Is it accompanies by a clear rights statement, detailing any copyrights? (from ODI certificates)
Is the rights statement machine-readable? (from ODI certificates)

26) Openly licensed
Is the standard openly licensed & legally reusable (= “open”)? (from ODI certificates)