What are the gaps, biases, and limitations within existing meta-analyses in the organochlorine pesticide literature?

Download PDF Download attached files

Attached Files:
ROSES for Systematic Map Protocol.xlsx
ocp_srm_search_benchmark.docx
ocp_srm_string_development_strategy_v2.xlsx
ocp_srm_data_coding_strategy.xlsx

Systematic Map Protocol

Title
What are the gaps, biases, and limitations within existing meta-analyses in the organochlorine pesticide literature?

Citation:
Losia Lagisz, Coralie Williams, Shinichi Nakagawa, Kyle Morrison. What are the gaps, biases, and limitations within existing meta-analyses in the organochlorine pesticide literature?: a Systematic Map Protocol. PROCEED-22-00043 Available from:
https://www.proceedevidence.info/protocol/view-result?id=43
https://doi.org/10.57808/proceed.2022.8

Corresponding author’s email address
z5393783@ad.unsw.edu.au

Keywords
Systematic Map, Organochlorine, Pesticide, Meta-analysis, Chemical Pollution

Background
The organochlorine pesticide literature is extremely diverse spanning a plethora of disciplines including ecological, biomedical, agricultural, environmental, and toxicological research (Peterson et al., 2017). Such diversity along with the rapid rise in literature makes keeping up with the primary research extremely difficult, particularly for policymakers and stakeholders. Consequently, secondary review studies have become increasingly useful to inform decisions by providing summaries of evidence (Squires, Valentine and Grimshaw, 2013). Since the 1990s, meta-analytic reviews have been used to provide quantitative evidence of the impacts chemical pesticides are having on the planet, becoming the keystone of reliable secondary research (Gurevitch et al., 2018). Research synthesis, particularly meta-analysis, allows reviewers to create rigorous summaries of many studies on the same topic. However, some topics may attract a lot of research synthesis efforts and others may not, which may limit our understanding of the broad patterns in the field. Additionally, the great reliance on meta-analysis as a powerful statistical tool comes with great responsibility. Poor methodical rigor and inappropriate statistical methods in the meta-analysis can lead to biased and misleading conclusions. Despite the potential risk of poor conduct in meta-analysis, there is currently no indication of methodical quality within the current meta-analysis investigating the impacts of organochlorine pesticides. Therefore, we propose a systematic review map that will identify the gaps, biases, and limitations of quantitative research synthesis within the organochlorine pesticide literature.

Theory of change or causal model
The use of meta-analysis within toxicology is ever-increasing which includes research on organochlorine pesticides. However, without a widespread understanding of where and how meta-analysis is carried out implementation of meta-analysis remains challenging. The proposed research is expected to quantify the gluts and gaps in organochlorine pesticide literature and provide scope for future research synthesis by identifying poor methodical rigor in past meta-analytic studies.

Stakeholder engagement
N/A

Objectives and review question
The objective of this systematic review map is to identify the gaps, biases, and limitations of current meta-analyses in the organochlorine pesticide literature. Primary questions: 1) What are the current research patterns of existing meta-analytic studies within the organochlorine pesticide literature? (i.e., which pesticides, subjects, and impacts have been synthesized meta-analytic studies). 2) What are the current methodological practices used in meta-analyses on the impacts of organochlorines? (e.g., which software, searching strategies and statistical models are mostly used in organochlorine pesticide meta-analysis). Secondary questions: 1) How are authors, studies, and keywords interconnected within the organochlorine pesticide literature?

Definitions of the question components
Population: Humans, non-human animals, and the environment (i.e., soil, water, and air). Exposure: Exposed to an organochlorine pesticide for example DDT, HCH, chlordane, and endosulfan (note: organochlorine pesticides are not limited to these pesticides). Comparator: Impacts of organochlorine pesticides on human, non-human animal, or environmental health compared to a control group that is not exposed to organochlorine pesticides. Outcome: All impacts on human, non-human animals, and environmental health. Study design: Meta-analysis and quantitative secondary research that contains meta-analysis. Timeframe: Any publication date.

Search strategy
The proposed systematic map methodology will follow an established method (James, Randall and Haddaway, 2016). The string development was completed on Scopus and is provided in section 8.1 below. We intend to use multiple databases to look for relevant studies including Web of Science, PubMed, ScienceDirect, and Cochrane Reviews. Searches will be restricted to publications and abstracts written in English. In addition, we have searched for grey literature using the BASE web based search engine. Syntax of the search strings will be customised for each database, as needed. For each database search, we will export citation information, bibliographic information, and abstracts. Additionally, we have searched for studies using a backward/forward citation search using relevant studies (see Additional file 1). The search results (bibliographic records) will be combined, and the duplicates will be removed in R. Finally, we will upload the deduplicated records to Rayyan (Ouzzani et al., 2016) for abstract screening. A total number of 4629 were found across all search methods combined (i.e., across all listed bibliometric databases, grey literature search, and forward/backward citation search). After deduplication, there was a total number of 3439 studies were found across all literature search methods.

Bibliographic databases
A search string has been designed for the Scopus database and completed on 4/8/2022 (see Additional file 1 for search string development strategy). Search strings for additional databases (i.e., Web of Science, Pubmed, Cochrane, and Science Direct) are also provided in Additional file 1. TITLE-ABS_KEY ( ( organoch* OR aldrin OR chlordane OR chlordecone OR dicofol OR dieldrin OR endrin OR heptachlor OR hexachlorocyclohexane OR hch OR lindane OR mirex OR pentachlorobenzene OR pentachlorophenol OR toxaphene OR ddt OR dichlorodiphenyltrichloroethylene OR dde OR dichlorodiphenyldichloroethylene OR endosulfan OR oxychlordane OR isobenzan OR isodrin OR ddd OR dichlorodiphenyldichloroethane OR methoxychlor ) AND ( ( meta* W/3 anal* ) OR ( systematic W/3 review ) OR ( scoping W/3 review ) OR ( realist W/3 review ) OR ( meta* W/3 regression ) OR ( comprehensive W/3 review ) OR ( meta* W/3 synthe* ) OR ( quantitative W/3 review ) OR ( quantitative W/3 synthe* ) OR ( global W/3 synthe* ) ) )

Web-based search engines
A search string was designed for BASE on 4/10/2022 to search for grey literature. A total of 38 studies were provided by the following search string: systematic* AND organochl* doctype:(14 18*)

Organisational websites
N/A

Comprehensiveness of the search
The search terms for the proposed systematic map were established through several rounds of discussion and testing. To assess the comprehensiveness of the search we completed a 10-study benchmark (Additional file 3), made of relevant studies from Google Scholar. We then tested whether the benchmark studies were found by the Scopus search string. This strategy was based on an already established method (Livoreil et al., 2017).

Search update
If more than 2 years pass between search and publication submission an update will be performed.

Screening strategy
The studies returned from the database searches will be screened in Rayyan (Ouzzani et al., 2016). Each study will undergo screening, firstly on titles and abstracts, and secondly on full texts. If a reviewer is unsure whether a study fulfils the title and abstract screening criteria the study will proceed to full-text screening (with Rayyan decision label maybe “maybe”). Further, in cases when the abstract is not available the study will be included in full-text screening (with Rayyan decision label maybe “maybe”). During the full-text screening, the reasons for each study rejection will be recorded. If the full text is unavailable or inaccessible the study will be rejected. To be included in the systematic map all full-text screening criteria must be met.

Eligibility criteria
Eligible population: We will include any study that investigates human, non-human animal, and environmental health. Eligible exposure: The study must assess the impacts of at least one organochlorine pesticide. Some common examples of organochlorine pesticides are DDT, HCH, chlordane, endosulfan, and dieldrin (note organochlorines are not limited to these pesticides). Eligible Comparator: The burden of organochlorine pesticides on health must be compared to a control group that either has less organochlorine pesticide exposure or no organochlorine pesticide exposure. Meta-analyses which assess the correlation between organochlorine pesticide exposure and health will be included. Eligible outcomes: The study must assess the impacts of organochlorine pesticide exposure on health. We will exclude studies investigating pesticide resistance, economic burden, pesticide alternatives, or policies on the use of organochlorine pesticides. Eligible Study design: The study must assess the impacts of at least one organochlorine pesticide using a meta-analysis (or equivalent quantitative method). Meta-analysis may be completed within other review types such as systematic reviews and global synthesis/reviews. We will accept studies that do not use conventional meta-analytic models but conduct quantitative synthesis across multiple studies or databases. Eligible timeframe: There will be no limit on the timeframe.

Consistency checking
All studies will undergo duplicate screening to check the consistency. Conflicts between reviewers will be resolved through discussion, if a conflict is persistent, we will consult with a third reviewer. Consistency between reviewers will be measured as a percentage of conflicts in screening decisions. A pilot screening of both titles and abstracts, and full texts has determined the consistency of screening outcomes between reviewers. A total number of 200 studies were screened across two pilots. Each pilot study consisted of 100 studies that were screened by two reviewers. The abstract screening of the first pilot showed a conflict rate of 2% between KM and CW, and the second pilot had a conflict rate of 3% between KM and ML. Each conflict was resolved through discussion. A total number of 48 successfully fulfilled abstract screening criteria (39 “maybe”, 9 “yes”) within the two pilots. Furthermore, pilot full-text screening revealed a 3% conflict rate between KM and CW in the first pilot set, and a 0% conflict rate between KM and ML in the second pilot set. Each conflict was resolved through discussion. A total number of 5 studies in the first pilot and 6 studies in the second pilot successfully fulfilled full-text screening criteria, out of a total 200 studies. Therefore, we can provide an estimation of 175-210 studies that will be included in the systematic reviews map.

Reporting screening outcomes
A record of each of the studies excluded and the reasons for exclusion will be provided as supplementary material. Additionally, a ROSES flow diagram will be used to record the number of excluded studies during each stage of screening. The ROSES flow diagram will also be provided as supplemental material. All deviations from this protocol will be reported within the formal systematic map.

Study validity assessment
A critical appraisal of each included meta-analysis will be completed using CEESAT (Woodcock, Pullin and Kaiser, 2014). This will indicate reporting quality of the existing meta-analyses in the organochlorine pesticide literature.

Consistency checking
During a pilot critical appraisal, we will discuss our interpretation of each CEESAT question by apprising in duplicate a random subset of 20% of accepted studies. Any differing opinions between reviewers will be discussed between them, if the conflict persists a third reviewer will be consulted. If the duplicate critical appraisal reveals a conflict rate of higher than 10% another random subset of 10% will be appraised.

Data coding strategy
All studies which fulfill abstract and screening criteria will undergo data extraction. Data extraction criteria have been based upon predefined criteria specified in Additional file 4. The extraction criteria have been designed to fulfil each of the primary and secondary questions of the review (see section 7).

Meta-data to be coded
The meta-data to be extracted can be found in the Additional file 4.

Consistency checking
Approaches for data extraction have been discussed during pilot extraction. A random subset of 20% of accepted studies will undergo duplicate data extraction. Conflicts between reviewers will be discussed between them, if an agreement cannot be made a third reviewer will mitigate it. If the duplicate extraction reveals a conflict rate of higher than 10% another random subset of 10% will be extracted in duplicate.

Type of mapping
A written report will coincide with the database (systematic map) to document the methods and results. The extracted data, meta-data, and code will be provided as supplementary material, as well as all details needed to replicate the data search and data screen.

Narrative synthesis methods
The systematic reviews map will present a narrative report which will include: i) Background and rationale, ii) details of the methodology, iii) critical appraisal of included studies using CEESAT, iv) recommendations for future research synthesis due to gaps in knowledge or poor methodical rigor and, v) implications for research, policy, and practice. Furthermore, the results of the systematic map will be presented in tables, plots, and heat maps.

Knowledge gap identification strategy
Gaps will be identified based on the distribution of the extracted data across the pre-defined categories, described in meta-data. Categories without any meta-analytic studies mapping onto them will be considered as potential gaps. Categories with many meta-analytic studies mapping onto them will be considered as potential gluts. The conclusions of the proposed systematic map will not only indicate the gaps and gluts, but also biases and methodological limitations of the existing meta-analysis on the organochlorine pesticide literature. The trends revealed by the systematic map will provide stakeholders, policymakers, and the scientific community with insight as to how and where current meta-analysis is carried out within the organochlorine pesticide literature.

Demonstrating procedural independence
Authors will not screen or extract any publication in which they are authors, or they assisted in the publication process.

Competing interests
The authors declare there are no competing interests.

Funding information
N/A

Author’s contributions
The research objectives, questions, and conceptualization were completed by KM under the supervision of SN and ML. Pilots were carried out by KM and duplicate pilots were carried out by CW and ML. KM wrote the first draft and lead the revisions. All authors contributed to the revisions of the protocol.

Acknowledgements
N/A

References
Gurevitch, J. et al. (2018) ‘Meta-analysis and the science of research synthesis’, Nature, 555(7695), pp. 175–182. Available at: https://doi.org/10.1038/nature25753. James, K.L., Randall, N.P. and Haddaway, N.R. (2016) ‘A methodology for systematic mapping in environmental sciences’, Environmental Evidence, 5(1), p. 7. Available at: https://doi.org/10.1186/s13750-016-0059-6. Livoreil, B. et al. (2017) ‘Systematic searching for environmental evidence using multiple tools and sources’, Environmental Evidence, 6(1), p. 23. Available at: https://doi.org/10.1186/s13750-017-0099-6. Ouzzani, M. et al. (2016) ‘Rayyan—a web and mobile app for systematic reviews’, Systematic Reviews, 5(1), p. 210. Available at: https://doi.org/10.1186/s13643-016-0384-4. Peterson, E.K. et al. (2017) ‘Integrative behavioral ecotoxicology: bringing together fields to establish new insight to behavioral ecology, toxicology, and conservation’, Current Zoology, 63(2), pp. 185–194. Available at: https://doi.org/10.1093/cz/zox010. Squires, J.E., Valentine, J.C. and Grimshaw, J.M. (2013) ‘Systematic reviews of complex interventions: framing the review question’, Journal of Clinical Epidemiology, 66(11), pp. 1215–1222. Available at: https://doi.org/10.1016/j.jclinepi.2013.05.013. Woodcock, P., Pullin, A.S. and Kaiser, M.J. (2014) ‘Evaluating and improving the reliability of evidence syntheses in conservation and environmental science: A methodology’, Biological Conservation, 176, pp. 54–62. Available at: https://doi.org/10.1016/j.biocon.2014.04.020.

Authors and Affiliations

Name	Country	Affiliation
Losia Lagisz	Australia	– Ecology and Evolution Research Centre, University of New South Wales, Sydney, Australia
Coralie Williams	Australia	Ecology and Evolution Research Centre, University of New South Wales, Sydney, Australia
Shinichi Nakagawa	Australia	– Ecology and Evolution Research Centre, University of New South Wales, Sydney, Australia
Kyle Morrison	Australia	Ecology and Evolution Research Centre, University of New South Wales, Sydney, Australia

Submitted: Sep 22, 2022 | Published: Oct 14, 2022

© The Author(s) 2022.
This is an Open Access document distributed under the terms of the Creative Commons Attribution 4.0 International License https://creativecommons.org/licenses/by/4.0/deed.en .