2021 Survey Methodology

On this page:

Survey Design

The survey instrument assessed employees’ awareness of the Scientific Integrity Policy and their experiences related to the culture of scientific integrity at EPA. Existing questions from both the 2016 and 2018 surveys were used as well as modified, and new questions were drafted to better assess aspects of scientific integrity and how the culture can be enhanced. Skip logic was incorporated into the survey instrument. Skip logic is a feature that allows personalization of the survey such that participants could only answer questions that were applicable to them based on their organization and/or position. This directed participants to parts of the survey that were applicable to them, reducing participant burden and a degree of measurement error.

The draft survey was pre-tested among nine experts within the Scientific Integrity Committee. Pre-testing is an important first step to address survey content, identify any problematic areas with question wording or interpretation, and reduce measurement error and respondent burden. The final survey was then entered into the Qualtrics online survey* management program and pilot tested among six external/non-EPA experts to identify any potential problems with the survey administration from start to finish. Each expert was instructed to complete the survey while assessing the clarity of instructions, ease of administration, item wording, and the acceptability of formatting. All feedback received was then incorporated into the final survey instrument. The final survey instrument was then approved for internal distribution.

Data Collection Procedures

The Qualtrics online survey* management program was utilized to conduct the survey. To ensure both anonymity and confidentiality, all eligible EPA federal employee email addresses (n=14,734) were imported into the Qualtrics* program. From Qualtrics*, a personalized email was sent to each federal employee with a personalized link, inviting their participation in the survey. To verify user information, once the link was accessed, input of the respondent’s email address was required as a crosscheck for eligibility. This verification process ensured that only Federal employees were participating in the survey and provided anonymity. Once the survey was started the Qualtrics* program disassociated the email address from the respondent, ensuring that survey responses could not be traced to an individual employee.

The online survey was open for three weeks and distributed to all current EPA employees from May 19^th to June 9^th, 2021. Weekly reminders were sent via Qualtrics* to those who either partially completed or did not start the survey and a final reminder was sent to everyone one day before the survey closed.

Response Rate

The survey was sent to all EPA Federal employees (n=14,734); this pool excluded contractors, grantees, fellows, students, volunteers, or any special appointments. Initially, due to the Agency firewall and cybersecurity, many employees experienced challenges with receipt of the external Qualtrics* survey invitation. These challenges were corrected; however, this inconvenience may have deterred participation and influenced the response rate. Approximately 4,470 (30.4%) accessed the link and started the survey and among these 2,668 employees (59.3%) completed the survey and submitted their responses. The table below reflects the participation, completion and overall response rate.

Total Eligible	14,734
Participation rate (surveys opened)	4479/14,734 (30.4%)
Completion rate (surveys submitted)	2668/4479 (59.6%)
Response rate	2668/14,734 (18.1%)

Representativeness

All federal employees were eligible for participation and had an equal opportunity to be a part of this survey. The sample used in the subsequent frequencies are representative of the larger population of EPA as the respondents’ demographic characteristics mirrors those of the Agency at the time this survey was distributed, using EPA’s May 2021 Diversity Dashboard as reference (table below).

Demographics	*2021 Survey*		*EPA*
Grade/classification level	*GS-8 or below*
	2.2%		3.0%
	*GS 9-11*
	9.1%		9.0%
	*GS 12 or 13*
	48.6%		53.0%
	*GS-14*
	17.5%		19.0%
	*GS-15*
	14.9%		15.0%
	*Other*
	0.9.0%		0.7.0%
Supervisory status - 1^st line, 2^nd line, other management levels	15%		13%
Program Office/Region	OA
	2.8%		2.3%
	*OAR*
	9.0%		7.9%
	*OCFO*
	1.3%		2.2%
	*OCSPP*
	9.2%		7.5%
	*OECA*
	2.7%		4.4%
	*OGC*
	1.3		1.6%
	*OIG*
	1.0		1.9%
	*OLEM*
	2.9		3.2%
	*ORD*
	16.5%		10.5%
	OW
	5.4%		4.0%
	*OITA*
	0.4%		0.4%
	*OMS*
	3.1%		7.2%
	*Region 1*
	3.1%		3.6%
	*Region 2*
	5.7%		5.2%
	*Region 3*
	3.7%		5.0%
	*Region 4*
	5.5%		5.9%
	*Region 5*
	6.4%		7.0%
	*Region 6*
	4.2%		4.6%
	*Region 7*
	3.6%		3.4%
	*Region 8*
	3.1%		3.5%
	*Region 9*
	4.4%		5.0%
	*Region 10*
	4.5%		3.5%
Racial and Ethnic Categories	*American Indian/Alaskan Native*
	0.4%	1.0%
	*Asian*
	5.4%		8.1%
	*Black/African American*
	8.8%		17.2%
	*Hispanic/Latinx*
	7.3%		8.0%
	*Native Hawaiian/Pacific Islander*
	0.1%		0.1%
	*White*
	69.8%		65%
	*Multiple categories***
	5.6%		1.0%
Sex	*Female*
	45.4%			52.9%
	*Male*
	42.4%			47.1%
Sexual Orientation	*Lesbian/gay*		data not available for comparison on EPA’s Diversity Dashboard
	2.9%
	*Straight*
	78.7%
	*Bisexual*
	2.1%
	*Something else*
	1.9%
Gender Identify	*Female*
	41.7%
	*Male*
	44.7%
	*Transgender*
	0.2%
	*None of these*
	1.9%

Survey Instrument

Survey Instrument (pdf) (336.45 KB)

*Disclaimer

Mention of or referral to commercial products or services, and/or links to non-EPA sites does not imply official EPA endorsement of or responsibility for the opinions, ideas, data, or products presented at those locations, or guarantee the validity of the information provided.