D. Quality measurement and reporting

D.  Quality measurement and reporting

19.37.    Quality management implies that countries undertake steps to measure quality and report the results of such measurements to all participants of the statistical process and general public. In that connection, the present Guide advises that countries develop a standard for regular quality reports that cover the full range of statistical processes and their outputs, including the publication of statistics on resident/non-resident transactions in services and FATS and additional indicators on the international supply of services, as applicable. Such reports can be either producer-oriented, with the aim of identifying strengths and weaknesses in the statistical process and leading to or containing the definition of quality improvement actions, or user-oriented, with the aim of keeping users informed about the methodology of statistical process and the quality of statistical output. 

19.88.    Quality reports should be prepared at least every five years, or more frequently if significant methodological changes or changes in the data sources occur. For monitoring the quality of the processes and the effectiveness of quality-improvement actions, reviews should be conducted more frequently. It is good practice for countries to base their quality reports on a set of quantitative and qualitative indicators, as well as on a checklist covering data collection, processing and dissemination, in order to assess the strengths and weaknesses in the statistical process and to identify possible quality-improvement actions.

19.39.    Countries are advised to develop their own quality assessment frameworks on the basis of NQAF and the quality assessment frameworks developed by international, supranational and regional organizations. 

19.40.    Quality measures Quality measures directly reflect a particular aspect of quality. For example, the time lag from the end of the reference period to the release of particular data set is a direct quality measure. However, in practice, quality measures can be difficult or costly to calculate. Instead, quality indicators can be used in the quality assessment.   

19.41.    Quality indicators  Quality indicators are summarized quantitative or qualitative evidences about the quality of the data. They are generally defined with respect to some reference point and can assist in making different types of comparisons. When countries define the quality indicators for their statistics on the international supply of services, they are encouraged to ensure that the indicators satisfy the following criteria: (a) they cover all dimensions of quality as defined in section B above, (b) they are based on the consistent application of a sound methodology and (c) they are easy to interpret both for internal and external users. 

19.42.    It is good practice for countries to maintain a balance between different dimensions of quality and the number of indicators. The objective of quality measurement is to have a limited set (minimum number) of indicators that can be used to measure and to follow over time the quality of statistics compiled within the framework for describing the international supply of services and to ensure that users are provided with a useful summary of overall quality, while not overburdening respondents with demands for unrealistic amounts of quality metadata. 

19.43.    Suggested quality measures and indicators  Table 19.1 presents a possible set of indicators and measures that countries might wish to consider for measuring the quality of statistics on the international supply of services. The table has been compiled on the basis of the quality measures and indicators recommended by various international organizations including IMF, OECD and Eurostat. Such measures and indicators provide users with a clear and up-to-date overview of the overall quality of those statistics.                                          

19.44.    One recognized good practice in relation to continuous quality improvement is process measurement. A process is a series of actions or steps towards achieving a particular end; process quality is an assessment of how well each step meets defined criteria; and process variables are factors that can vary with each repetition of the process. It is good practice for producers of official statistics to define and produce a selection of process quality measures to provide an indication of the overall quality of processes and facilitate continuous quality improvement. 

19.45.    For official statistics produced from surveys, examples of process quality measures are the percentage of ineligible sampling units found in the sample, the proportion of proxy interviews by survey, travel time for interviewers and scanning/keying error rates. 

19.46.    When dealing with statistics produced from administrative sources, process quality measures may be the number of queries from the statistical producer to the administrative data supplier, and the percentage of data items changed during quality assurance. 

19.47.    When adopting new or improved methods, quality assurance procedures should be used. They should include assessing the impact on the statistical series of adopting the new methodology and subjecting the proposed methodology to peer review. Various mechanisms exist to gain input from experts into the suitability of new or improved methods. The setting up of specific peer review groups and collaboration with users, academics and subject matter experts, for example industrialists or demographers, can also be considered or simply peer review from a statistical colleague.^[2] 

19.48.    The quality criteria as described in the present chapter also apply to statistics on the international supply of services by mode.

Next: E. Country experiences (Chapter 19)