Forms: Creating an Effective Survey

This page provides information and details regarding the two preferred survey tools at NYSERDA: Microsoft Forms and Qualtrics, along with other important information you should know before attempting to create a survey with either of these tools.

Consult your evaluation liaison if you need help to develop a complex survey or if you have questions regarding survey design.
Public facing surveys should always be reviewed by the Marketing team and your Program Attorney before being deployed.
(The Marketing review focuses on evaluating comprehension and customer experience and will engage the Communication team as needed.)

Process Overview

The below steps outline the high-level process when creating a new survey. Click each tab to learn more about the actions that should be taken in each major step of the process. Should you have any questions about the information on this page, or when developing a survey, consult with your evaluation liaison.

Outline the requirements for your survey.

What is the purpose or goal of the survey?
What data/information are you looking to capture and what will it be used for?
Who will be responding to the survey?
Are the potential respondents internal or external stakeholders?
Are there other colleagues at NYSERDA who could benefit from the data collected? If yes, include them in the survey development.

Determine the Type of Survey that will best meet your needs.

Survey Type

Description

Questionnaires

Are usually completed independently by respondents. Questionnaires are more likely to return Quantitative data. Quantitative data can be counted, measured, and expressed either categorically or numerically.

example: communicated poll utilizing a rating scale or closed-ended questions (yes/no) to determine who can attend an upcoming meeting.

Interviews

Are generally conducted by an interviewer to document participant responses. Interviews are more likely to return Qualitative data. Qualitative data is descriptive, conceptual and can be categorized based on traits or characteristics.

example: research conducted using open-ended questions in a focus group environment to determine user experience efficacy for a new application.

Evaluate how long the survey should be.

There is a delicate balance between asking enough questions to gather all needed data while minimizing how long a participant spends responding to a survey. Too few questions and you might not obtain the insights you're looking for and too many questions can alienate the participants, leading to incomplete surveys and less reliable data.
In general, a Questionnaire utilizing mostly close-ended questions should take no more than 20 minutes for a participant to complete. Interviews can take longer based on the nature of the questions and the answers provided.
There are many factors that can help determine how long a respondent will be engaged in a survey, such as:
- The survey method used.
- The type of questions you'll be asking (open versus closed-ended).
- Possible incentives offered, either real or perceived.
- The frequency of survey activity.

Determining how long you'd want a participant to spend on the survey can help outline what types of questions you need to ask and how they should be worded to avoid confusion.

**Note: While there is no perfect survey length, it is more important to estimate the desired participant response time first, and then generate the required questions according to that scale, rather than the other way around.

Outline the Survey Questions.

Question Categories:

Question Category

Description and Examples

Pros

Cons

Open Ended Questions

Allow respondents to answer the questions in their own words and usually provide a text box for their unique input.

example: What do you think about the program?

answer:

test test test test test testtest test test test test test

Easier to write
Allows for more detail
May produce unexpected answers
More salient

More difficult to answer (greater burden on respondent)
Must code answers (interpret & categorize responses)
Harder to analyze

Close Ended Questions

Require respondents to select their response from a provided list of options

example: Where did you hear about the program?

answer options:

Website
Utility Bill
Tv or Radio Advertisement
Friends or Family

Less expansive
Easier to analyze
Can test specific responses

Harder to write
Need a complete list of responses
Limited number of responses

Question Types (options in the below list are available in both tools):

Question Type	Description
Forced Choice	Multiple options to choose from where only one can be selected.
Select-All Choice	Multiple options to choose from where all options that apply can be selected
Date	Provides ability to enter in or select a date.
Likert	Generally a 5 or 7 point scale which allows a respondent to express how much they agree or disagree with a particular statement.
Ranking	Asks respondents to compare items to each other by placing them in order of preference.
Rating	Displays a scale of answer options from any range (0 to 100, 1 to 10, etc.) where respondents select the number that most accurately represents their response.
Text	Allows respondents to enter in a single line or multiple lines of text.

**Note: The above table lists all available questions types that are available with Microsoft Forms. Qualtrics, however, has over 100 possible questions type, most of which are not displayed in the table above.

Branching:

Branching is a more complex survey function that allows a different set of questions to be shown to respondents based on how they answered the previous question. This function is available in both Microsoft Forms and Qualtrics and should be used with care as the branching logic needs to be well thought out and implemented correctly for it to be an effective tool.

Finally, determine the best survey tool for your purposes.

The chart below provides some high level information regarding each of the tools available to aid in your decision.

	Microsoft Forms	Qualtrics
When to Use	Great for internal or simple and short external surveys. ex: Survey to be distributed to NYSERDA staff ex: Simple poll to assess external partner meeting availability	Great for complex external surveys. ex: Online accessible survey link posted for public engagement ex: Survey requesting external contacts to respond with requested file attachments
Ease of Use	Easy to use application with simple, essential controls that allows for quick and agile survey creation and communication.	A powerful online survey tool with robust functionality that may require training or guidance to learn how to use.
How to Access	Available to all NYSERDA staff with an active Office365 license.	Any NYSERDA staff member can request a Qualtrics account by email Tricia Gonzales at patricia.gonzales@nyserda.ny.gov.
How to Use		Qualtrics Login After logging into the above link, you can access the training entitled Quick Start with Qualtrics that provides a good overview of the application functionality.

Both tools offer the following additional functionality:

Collaboration with colleagues and stakeholders during survey development.
Data export to Microsoft Excel for analysis (this also includes automatic updates for Excel exports when new responses are submitted:
- Qualtrics supports real-time updates.
- Microsoft Forms supports batched updates requiring users to access the Microsoft Excel export and clicking the Refresh All option in the Data menu of the workbook.
Can be used to create polls or surveys

Create the survey introduction.

You survey introduction should include/consider the following:

Brief description of the survey effort, goals, objectives and/or long-term outcomes.
How the survey benefits the respondent (i.e., why is it important that they complete this survey).
Estimated amount of time it will take a participant to complete the survey.
How and where the data will be used.
If the survey responses will be kept anonymous.
How long participants have to respond to the survey before the response period ends.
Contact name, email and phone number of NYSERDA staff at the top of the introduction to help participants verify the legitimacy of the survey.
Contact name, email and phone number at the bottom of the introduction for participants if they require technical assistance or if they have questions (can be the same contact as above).

Disclaimer

NYSERDA is not able to promise complete confidentiality of survey answers because we are a Public Authority subject to FOIL. If your survey responses should be treated as confidential, please include the following language: The information you provide will be kept confidential to the extent permitted by law including, but not limited to, the Freedom of Information Law (FOIL).

Document your questions in the survey tool.

For each question in a survey, consider how well it addresses what you are trying to measure. Content-related questions you can use to determine the efficacy of your survey questions are:

Is the question absolutely necessary to ensure you obtain useful information?
Are several questions needed or will one suffice instead?
Do participants have all the information needed to accurately and complete answer the questions?
Is the question specific or general enough?
Does the question exhibit a tendency to be biased or loaded?
Is the question worded in a way that will prevent the participant from answering truthfully?

Evaluate the answer choices, placement and rating scale of your questions.

Answer Choices:

Possible participant response options should be exhaustive and mutually exclusive.

Question Placement:

The first few questions set the tone for the survey and can help put your respondent at ease.
More sensitive or difficult questions should appear later in the survey.
Try to ensure that possible answers to questions are only influenced by previous questions intentionally, such as when deploying branching.
Rearrange questions as needed to encourage a positive experience and garner interest in participants to respond to all questions.
Try to find the flow in the survey so that participants feel encouraged to continue.

Question Rating Scales:

1-5 and 1-7 are optimal rating scale ranges.
Always include descriptive labeling for each increment in the rating scale range.
Include a neutral mid-point by employing an odd rating scale range, rather than an even one to prevent bias in the rating scale.

Evaluate the efficacy of your questions.

What to Avoid	Examples
Multiple Subject Questions	Double-barreled Questions are questions that contain two arguments/subjects. example: Are you satisfied or dissatisfied with the audit program sign-up process and the time it took to schedule your audit? Double-negative Questions are questions that employ two negative words such as no and not. example: Do you agree that people who save energy is not uncommon?
Response Option Overlaps	If you ask a question such as: example: How many LED bulbs did you purchase? Your response options should include response options such as: 0-5 6-10 11-15 16+ Rather than incorrect response options such as: 0-5 5-10 10-15 15+
Leading Questions	Questions that influence the way in which respondents answer your questions. example: You like NYSERDA’s new logo, don’t you?
Loaded Questions	Questions that suggest a socially desirable answer or can be emotionally charged. example: Is it terrible for people to waste limited resources?
Assumptions	Avoid questions that assume a respondents familiarity with specific subjects. example: Does a lack of courtesy shown by hotel receptionists, in your opinion, influence the customer experience of at hotels?
Jargon or Slang	Use words that are direct and familiar to the respondents.
Social Desirability Bias	People naturally want to appear as good, upstanding members of their community to others will sometimes distort their survey response to encourage that appearance. Survey questions about behaviors or attitudes that are socially-desirable or undesirable can suffer from response bias. Wording that may encourage Social Desirability Bias: example: I’m will read a list of actions you may have taken to reduce the amount of energy your household uses. After reading each one, please tell me if this is something you have done. Instead, try: example: In talking with people about ways they might reduce the amount of energy their household uses, we often find that people aren’t sure what actions will save them energy or they are too busy make changes in their routines. I’m will read a list of actions you may have taken to reduce energy consumption. After reading each one, please tell me if this is something you have done. Self-administered surveys, such as web-based questionnaires, are less likely to suffer from social desirability bias than a telephone or in-person survey.

Page tree