E.1 Audit Requirements

Audits seek to mirror institutional (customer) usage on a report provider’s platform. COUNTER defines specific audit tests for each of the required reports. As report providers may work with different auditors, the audit test scripts in this appendix help ensure each auditor follows a common auditing procedure, as well as ensuring the reports comply with the Code of Practice.

There are three aspects of audit compliance

  • Report delivery: Auditors test whether the COUNTER_SUSHI API has been correctly implemented for delivery of JSON reports, and whether the web interface permits retrieval of tabular reports, using the instructions supplied by the report provider.

  • Format: JSON and tabular reports are reviewed by the auditor to confirm they adhere to the format and presentation specified in the Code of Practice.

  • Data integrity: Metrics included in the reports are verified by the auditor to ensure they accurately record activities carried out during audit seeding. This includes

    • Checking that reports are consistent regardless of the browser used for seeding and/or accessing reports. Auditors MUST use a minimum of two different browsers for testing, one of which MUST be Chrome, Edge, Firefox, or Safari. Any browser used for testing must account for a minimum of one quarter (25%) of the overall testing.

    • Verifying whether the report provider operates a cache server, as this may result in under-reporting of seeding activity. Auditors MUST always switch off the browser cache settings of test machines.

Note that auditors are not obliged to express an opinion regarding usage reported in any other accounts/institutions, or regarding aspects of the Code of Practice not specifically tested.

Audit test requirements vary depending on the set up of the platform and any related database(s), as indicated by the Options within the tests below. For each applicable audit test to achieve an audit pass, all metrics reported MUST be within a -8% and +3% reliability window of the same audit metrics on the auditor’s report.

E.1.1 Auditor Setup

The reporting period for COUNTER is the calendar month: a report run for any given month MUST reflect all activity of a customer for the entire selected month. For the purposes of audits, auditors MUST conduct and conclude all audit tests within the seeding month.

Prior to an audit, the report provider SHOULD supply to the auditor, using either the setup form provided (PDF) or the auditor’s preferred alternative:

  • Account details for at least four separate accounts with access to all areas of the site (or specific restrictions for testing Denials).

  • SUSHI credentials for the test accounts to enable verification of SUSHI harvesting of JSON format reports.

  • Access to download usage reports in tabular format.

  • A declaration confirming federated and automated searches have been disaggregated from any searches reported.

  • If server-side caching is implemented, information on cache settings used SHOULD be provided.

To prevent any collision of reported data, auditors MUST be allowed to set up and maintain separate accounts for each of the audit tests. During the seeding month, there MUST NOT be any activity on the audit accounts other than activity generated by the auditor. Any non-auditor activity on the test accounts will make the test reports unreliable and result in failure of the audit.

Unless otherwise specified the auditor MUST

  • Have access to all available content on the platform.

  • Allow at least 31 seconds between each metric-triggering action on the platform.

  • Record the database searched (if applicable).

  • Record the numbers of content items investigated or requested.

  • Perform each separate test within its own separate session. Tests that are not completed within a separate session will give incorrect results, particularly where testing unique metrics such as Unique_Item_Investigations and Unique_Title_Requests.

E.1.2 Audits Where ‘Accounts’ Do Not Apply

Institutional repositories typically cannot set up or supply account details for auditors. The same may be true for purely open access publishers. Where this is the case, all audit tests MUST be run on defined and agreed IP addresses.