B. Quality measurement and reporting

On this page

1.         Steps and guidelines for producing data quality reports

9.23.        Steps to be considered. Some information about data quality is available in all offices involved in the compilation and dissemination of official trade statistics and its compilation often provides the starting point for a quality report or might already be considered to be a quality report. The following considerations (or steps) that are suggested for the production of a data quality report for merchandise trade statistics also appear to be relevant when a quality report already exists but is being reviewed:

(a)        Collection and review of existing standards, guidelines, requirements, practices, examples or past quality reports within the same office or other offices within the same country or in other countries, including regional and international organizations, as appropriate, in order to ensure that standards and best practices are followed;

(b)   Discussion and decision on the scope and type (purpose) and frequency of the quality report under consideration and on the available resources;

(c)    Assembly of a team and allocation of resources;

(d)   Elaboration and discussion of the detailed structure of the quality report;

(e)    Compilation of the required information: quality assessment and measurement;

(f)    Drafting of report;

(g)   Review of report;

(h)   Dissemination and communication of the report and its results.

9.24.        Guidelines for the production of quality reports in the ESS. Within the European Statistical System (ESS), very strong efforts have been undertaken to develop a concept of quality and to implement it comprehensively. A central achievement was the adoption of the European Statistics Code of Practice, which provides a broad conceptual framework for viewing quality and sets standards for the institutional environment, statistical processes and statistical outputs. The ESS Standard for Quality Reports issued by Eurostat[11] provides recommendations for preparing comprehensive quality reports for the full range of statistical processes and their outputs. The ESS Handbook for Quality Reports[12] provides much more detailed guidelines and examples of quality reporting practices.

9.25.        Specific objectives of the ESS guidelines. The specific objectives of the ESS guidelines contained in the ESS Standard for Quality Reports are; (a) to promote harmonized quality reporting across statistical processes and their outputs within a country and hence to facilitate comparisons across processes and outputs; (b) to promote harmonized quality reporting for similar statistical processes and outputs across countries and hence to facilitate comparisons across countries; and (c) to ensure that reports include all of the information required to facilitate identification of statistical process and output quality problems and potential improvements.

9.26.        Structure of the ESS guidelines. The guidelines are organized by statistical output and process quality components, with the primary section headings being:

  1. Introduction to the statistical process and its outputs  (an overview to provide context)
  2. Relevance (an output quality component)
  3. Accuracy (an output quality component)
  4. Timeliness and punctuality (output quality components)
  5. Accessibility and clarity (output quality components)
  6. Coherence and comparability (output quality components)
  7. Trade-offs between output quality components
  8. Assessment of user needs and perceptions (covering all aspects of output quality)
  9. Performance, cost and respondent burden (process quality components)
  10. Confidentiality, transparency and security (process quality components)
  11. Conclusions (summary of principal quality problems and proposed improvements)

9.27.        ESS standard quality reports. The ESS standard quality reports are producer-oriented, as they have been primarily designed to assist EU member States in internal self-assessment and reporting to Eurostat. Nevertheless, as considerable emphasis is put on output quality, they include as well all the information necessary for user-oriented quality reporting.

2.         User-oriented quality reports: contents and examples

9.28.        Characteristics. User-oriented quality reports are keeping users informed about the methodology of the statistical process and the quality of statistical output. Many statistical agencies have adopted principles and standards for data quality and a data quality assessment framework which outlines the different dimensions of quality and their measurement. The quality assessment framework provides a general layout for the quality report; however, not all dimensions are equally relevant for users. User-oriented quality reports are often provided on an on-going basis as part of the metadata provided to users, and are updated regularly.

9.29.        Quality report of Germany.[13] The quality report for the foreign trade statistics of Germany consists of (a) general information about the statistics, such as name of statistics, reporting period, subject, respondents, legal framework and confidentiality; (b) purpose, as defined by data variables, its justification and users; (c) compilation methods, describing how the data are obtained and the burden on respondents; (d) accuracy describing coverage, customs and statistical threshold and estimations and revisions; (e) timeliness; (f) comparability with data of others and over time; (g) coherency, describing the relationship to related statistics; and (h)  references to additional information.

9.30.        United States merchandise trade statistics quality report.[14] The report entitled, “U.S. merchandise trade statistics: a quality profile” provides information on the quality of the statistical program, and is intended to aid data users in their understanding and appropriate use of the data. It addresses issues affecting the quality of statistics, and some known limitations. For example, undocumented export shipments were identified through comparisons with trade data of major trading partners, audits of trade documentation, and other measures. Implementing the data exchange withCanada (see para. 9.47 below) and mandatory electronic filing (see para. 9.16 above) has reduced these errors and improved data coverage.

9.31.        Eurostat merchandise trade statistics quality report. The report of Eurostat entitled Quality Report on International Trade Statistics provides data users with quality indicators and information regarding the practices of EU member States. It summarizes the main outcomes of the national quality reports that member States must provide to Eurostat each year within a fixed deadline. Its structure and contents follow the ESS guidelines for quality reports.

9.32.        European Statistical System: Euro-SDMX Metadata Structure. Applicable within the European Statistical System, the Euro SDMX Metadata Structure (ESMS) contains the description of statistical metadata concepts for documenting statistical data and for assessing data quality and the production process in general. With regard to its data quality components, the ESMS follow quality criteria in line with the European Statistical Law: relevance, accuracy, timeliness, punctuality, comparability, coherence, accessibility and clarity. Special attention is given to a set of quality and performance indicators aiming at quantifying the various quality criteria and at providing a common standard across the ESS. The ESMS documentation should accompany data dissemination of Eurostat and the EU member States.

9.33.        IMF SDDS on international merchandise trade.[15] Countries that subscribe to the International Monetary Fund (IMF) Special Data Dissemination Standard (SDDS) make a commitment to observe the Standard and to provide information about their data and data dissemination practices (metadata) on the IMF Dissemination Standards Bulletin Board (DSBB). One of the areas covered by the IMF SDDS is international merchandise trade statistics. The SDDS metadata are available in two presentations: the current SDDS format and the Data Quality Assessment Framework (DQAF) format which covers six dimensions: 0. Prerequisites of quality; 1. Assurances of integrity; 2. Methodological soundness; 3. Accuracy and reliability; 4. Serviceability; 5. Accessibility. The SDDS was established to guide members that have, or that might seek, access to international capital markets in the provision of their economic and financial data to the public. To date, there have been 68 subscriptions to the SDDS.

3.         Producer-oriented quality reports – contents and examples

9.34.        Characteristics. Producer-oriented quality reports aim at identifying strengths and weaknesses of the statistical process and lead to, or contain the definition of, quality improvement actions. Producer-oriented quality reports can be either motivated by internal interest or externally mandated. They can take the form of, for example, internal review, benchmarking (comparison with others) and audits. By their nature, producer-oriented quality reports are often produced for particular reasons, for example, to fulfill a specific external requirement or to deal with specific issues or problems.

9.35.        Individual assessment reports for EU member States.[16] An assessment is prepared annually by Eurostat for all EU member States based on their responses to a quality report, with the overall goal of achieving quality improvements. The assessment is structured according to the following quality dimensions: relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability and coherence, some of which are further subdivided. Under each of these dimensions, a set of items is listed along with the specification of their requirements. For example, under accuracy–coverage, the item “statistical threshold in value” should be below or equal to 1,000 euros in value and 1,000 kilograms in net mass. If an item is a legal requirement, it is evaluated according to a four-point rating scheme which ranges from 1 (“serious persistent infringement”) to 4 (“compliance”). If an item refers to a recommendation, the assessment can be: A (“fully applied”), B (“partially applied”) or C (“not applied”). Some items are not evaluated if they are not, or not yet, requirements.

9.36.        Mission reports of international and regional organizations. International or regional organizations can be requested by countries to conduct an assessment of the statistical programme in a particular area such as merchandise trade. Such requests have often the purpose of creating guidance on and the impetus to address institutional or general data compilation issues and will focus on the particular areas concerned. In other cases, assessments have been conducted as a part of technical assistance activities which aim at the overall application of the international recommendations for international merchandise trade statistics and accordingly focus on these concerns. As regards the United Nations Statistics Divisions, such assessments were always provided on the basis of the concepts and definitions for international merchandise trade statistics. The updated recommendations (IMTS 2010) provide a much better basis for such assessments than the previous recommendations, since, in contrast with the previous recommendations, the entire data compilation process is covered.

4.         Measuring data quality

9.37.        Use of quality measures and indicators. The measurement of the quality of any statistical data, including international merchandise trade statistics data, is not a simple task. Problems arise from the difficulties involved in quantifying the levels of individual quality dimensions and in aggregating the levels of all dimensions. Under these circumstances, deriving a single quantitative measure of quality is not possible. In the absence of such a single measure, countries are encouraged to use a system of quality measures and indicators (see IMTS 2010, para. 9.13).

9.38.        Quality measures and indicators. Quality measures directly reflect a particular aspect of quality. For example, the time lag from the end of the reference period to the release of international merchandise trade statistics is a direct quality measure. However, in practice, the calculation of quality measures can be difficult or costly. Instead, quality indicators may be used for the quality assessment. Quality indicators provide summarized quantitative or qualitative evidence of the quality of the data. They are generally defined with respect to some reference point and can assist in making different types of comparisons (ibid., paras. 9.14-9.15).

9.39.        Methods and tools for measuring data quality. Methods for measuring quality encompass documentation and reporting, the calculation of indicators, auditing procedures, self-assessment and questioning the users. The Eurostat “Handbook on data quality assessment methods and tools”[17] aims at facilitating a systematic implementation of data quality assessment in the European Statistical System. It presents the most important assessment methods: quality reports, quality indicators, measurement of process variables, user surveys, self-assessment and auditing, as well as the approaches of labelling and certification. The Handbook provides a concise description of the data quality assessment methods currently in use. Furthermore, it offers recommendations on how these methods and tools should be implemented and how they should reasonably be combined. The methods and tools presented in the handbook facilitate an assessment of statistical products, and statistics production processes, as well as the user perception of statistical products. Annex A of the handbook includes a background paper on the position of data quality assessment in the general framework of quality management and annex B presents examples of good practice in greater detail. The Handbook is primarily targeted towards quality managers.

9.40.        Information on quality measurement: United Kingdom of Great Britain and Northern Ireland. The following metadata on quality are provided to users of United Kingdom trade statistics: (a) quality standards against which the quality is measured and (b) Assessment (including self-assessment) against some of these quality standards, including (i) quantitative assessment against indicators for the six output quality dimensions, (ii) qualitative assessment of own methods and adherence to EU legislation and (iii) channels for and results of post-publication quality assurance.

 

 

11  See ESS Standard for Quality Reports: 2009 edition, Eurostat Methodologies and Working Papers (Luxembourg, Office for Official Publications of the European Communities, 2009).

12  See footnote 1 of chapter 9.

13  The information is derived from the following publication: “Qualitaetsberich Aussenhandel” (Wiesbaden, Germany, Statistisches Bundesamt, January 2011).

14  Washington, D.C., Bureau of the Census, Foreign Trade Division, Methods Research and Quality Assurance Branch, 19 December 2002.

15  See http://dsbb.imf.org/Pages/SDDS/Home.aspx.

16  See European Commission and Eurostat, “United Kingdom: 2009 Individual assessment report—final version”.

17  Manfred Ehling and Thomas Körner, eds. (Wiesbaden, Germany, European Commission, 2007).