How we evaluate guidelines

The Partnership rates records in the Cancer Guidelines Database using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument.

The AGREE II is used internationally by both guideline developers and users. It provides a framework to:

• Assess the methodological rigour and transparency in which a guideline is developed
• Provide a methodological strategy for the development of guidelines
• Inform what information ought to be reported in guidelines and how

How does the AGREE II work?

The AGREE II consists of 23 items targeting various aspects of practice guideline quality across six quality domains:

1. Scope and Purpose
2. Stakeholder Involvement
3. Rigour of Development
4. Clarity and Presentation
5. Applicability
6. Editorial Independence

Higher domain scores (percentiles) reflect greater guideline reporting quality. The AGREE II does not generate a global aggregate score. Each domain score is available in the Cancer Guideline Database for all guidelines that met the inclusion criteria.

For a guideline to receive high scores, there must be a clear link between the proper collection and use of research evidence by an appropriate and representative guideline panel, that uses a transparent and replicable process to develop recommendations and implementation considerations made in the guideline. The higher the AGREE II scores, the more confident you can be that the guideline developers used an evidence-based approach to reach their recommendations.

What AGREE II criteria are used for the Cancer Guidelines Database?

We evaluate guidelines proposed for inclusion in the Cancer Guidelines Database using the eight items in the Rigour of Development domain of the AGREE II. Those scoring 60% or above will be assessed using the full AGREE II Instrument. Guidelines scoring between 50% and 60% will receive an additional appraisal to confirm that the 60% cut-off is appropriate and is not unfairly excluding guidelines of reasonable quality.

The 60% threshold is based on the following sources:

• “No universal agreement exists about specific cut-off scores to identify high-quality guidelines. Some domains (e.g., rigour of development) may be considered more important than others and thus have a higher benchmark. The group should identify the range of acceptable-quality scores (e.g., ⩾70/100) and whether different domains should have different cut-offs. Guidelines meeting the benchmark for the ‘rigour of development’ domain can then be assessed using the other AGREE domains.” View the published article: EBN users’ guide: Evaluation and adaptation of clinical practice guidelines.
• “The CTF [Canadian Task Force on Preventive Health Care] has established that guidelines must achieve a score of at least 60% in the Scope and Purpose, Rigour of Development, and Editorial Independence domains to be considered high-quality rigorously developed guidelines.” View the published article: Recommendations on screening for high blood pressure in Canadian adults.
• Although there have been no definitive studies specifying what threshold clearly indicates a high-quality guideline, other groups have used the 60% threshold to indicate that guidelines score “favourably versus unfavourably,” or that domains with scores greater than 60% are considered “effectively addressed.” View the published articles:
  • A critical appraisal and comparison of the quality and recommendations of glaucoma clinical practice guidelines,
  • A Systematic Critical Appraisal for Non-Pharmacological Management of Osteoarthritis Using the Appraisal of Guidelines Research and Evaluation II Instrument,
  • A critical appraisal of guidelines for the management of knee osteoarthritis using Appraisal of Guidelines Research and Evaluation criteria
• The Guidelines Advisory Committee found that guidelines with a Rigour of Development score below 70% would not be of sufficient quality to receive Committee endorsement, and therefore used 70% as their quality cut-off for consideration. The purpose of the Cancer Guidelines Database is not to propose guidelines for endorsement, so a lower threshold is reasonable at this time.
• Brouwers et al developed “high quality” and “low quality” guideline examples for testing the items in the beta version of AGREE II. The average scores for the Rigour of Development domain were 78% for the high-quality guidelines and 55% for low-quality guidelines. View the published article: Development of the AGREE II, part 2: assessment of validity of items and tools to support application.
• On the AGREE II seven-point scale, an average score of just above the midpoint (5) would equal 66.6%. To allow for some flexibility in scoring for the Cancer Guidelines Database, 60% would serve as an acceptable quality threshold.

How can I use the AGREE II to develop guidelines?

Developing a guideline in accordance with the standards outlined in the AGREE II ensures that all the important aspects of guideline development have been performed and are explicitly reported.
If a guideline contains information from each domain, it should be helpful to end users. For more information please visit the AGREE Trust website.

How do you evaluate guidelines from organizations?

In addition to the guidelines that we search, filter and appraise, we evaluate guidelines developed by key Canadian and provincial cancer-focused organizations (identified by the Partnership) that satisfy the formal inclusion criteria. They undergo the Cancer Guideline Database’s full quality appraisal process, including evaluation using the AGREE II domain Rigour of Development, and full AGREE II appraisal if they meet the 60% Rigour of Development threshold.

In the records for these guidelines, we indicate:

• AGREE II scores are available. These are guidelines that meet the inclusion criteria and have achieved the threshold of 60% (37 and above) for Rigour of Development.
• AGREE II scores are not available.
  1. These guidelines either do not meet the threshold of 60% in the AGREE II rigour domain, or
  2. They are Canadian guidelines that do not meet the inclusion criteria.

View the database’s complete inclusion and evaluation criteria.