This page provides information and details regarding the two preferred survey tools at NYSERDA: Microsoft Forms and Qualtrics, along with other important information you should know before attempting to create a survey with either of these tools.
Consult your evaluation liaison if you need help to develop a complex survey or if you have questions regarding survey design.
Public facing surveys should always be reviewed by the Marketing team and your Program Attorney before being deployed. (The Marketing review focuses on evaluating comprehension and customer experience and will engage the Communication team as needed.)
Process Overview
The below steps outline the high-level process when creating a new survey. Click each tab to learn more about the actions that should be taken in each major step of the process. Should you have any questions about the information on this page, or when developing a survey, consult with your evaluation liaison.
Outline the requirements for your survey.
What is the purpose or goal of the survey?
What data/information are you looking to capture and what will it be used for?
Who will be responding to the survey?
Are the potential respondents internal or external stakeholders?
Are there other colleagues at NYSERDA who could benefit from the data collected? If yes, include them in the survey development.
Determine the Type of Survey that will best meet your needs.
Survey Type
Description
Questionnaires
Are usually completed independently by respondents. Questionnaires are more likely to return Quantitative data. Quantitative data can be counted, measured, and expressed either categorically or numerically.
example: communicated poll utilizing a rating scale or closed-ended questions (yes/no) to determine who can attend an upcoming meeting.
Interviews
Are generally conducted by an interviewer to document participant responses. Interviews are more likely to return Qualitative data. Qualitative data is descriptive, conceptual and can be categorized based on traits or characteristics.
example: research conducted using open-ended questions in a focus group environment to determine user experience efficacy for a new application.
Evaluate how long the survey should be.
There is a delicate balance between asking enough questions to gather all needed data while minimizing how long a participant spends responding to a survey. Too few questions and you might not obtain the insights you're looking for and too many questions can alienate the participants, leading to incomplete surveys and less reliable data.
In general, a Questionnaire utilizing mostly close-ended questions should take no more than 20 minutes for a participant to complete. Interviews can take longer based on the nature of the questions and the answers provided.
There are many factors that can help determine how long a respondent will be engaged in a survey, such as:
The survey method used.
The type of questions you'll be asking (open versus closed-ended).
Possible incentives offered, either real or perceived.
The frequency of survey activity.
Determining how long you'd want a participant to spend on the survey can help outline what types of questions you need to ask and how they should be worded to avoid confusion.
While there is no perfect survey length, it is more important to estimate the desired participant response time first, and then generate the required questions according to that scale, rather than the other way around.
Outline the Survey Questions.
Question Categories:
Question Category
Description and Examples
Pros
Cons
Open Ended Questions
Allow respondents to answer the questions in their own words and usually provide a text box for their unique input.
example: What do you think about the program?
answer:
test test test test test testtest test test test test test
Easier to write
Allows for more detail
May produce unexpected answers
More salient
More difficult to answer (greater burden on respondent)
Must code answers (interpret & categorize responses)
Harder to analyze
Close Ended Questions
Require respondents to select their response from a provided list of options
example: Where did you hear about the program?
answer options:
Website
Utility Bill
Tv or Radio Advertisement
Friends or Family
Less expansive
Easier to analyze
Can test specific responses
Harder to write
Need a complete list of responses
Limited number of responses
Question Types (options in the below list are available in both tools):
Question Type
Description
Forced Choice
Multiple options to choose from where only one can be selected.
Select-All Choice
Multiple options to choose from where all options that apply can be selected
Date
Provides ability to enter in or select a date.
Likert
Generally a 5 or 7 point scale which allows a respondent to express how much they agree or disagree with a particular statement.
Ranking
Asks respondents to compare items to each other by placing them in order of preference.
Rating
Displays a scale of answer options from any range (0 to 100, 1 to 10, etc.) where respondents select the number that most accurately represents their response.
Text
Allows respondents to enter in a single line or multiple lines of text.
The above table lists all available questions types that are available with Microsoft Forms. Qualtrics, however, has over 100 possible questions type, most of which are not displayed in the table above.
Branching:
Branching is a more complex survey function that allows a different set of questions to be shown to respondents based on how they answered the previous question. This function is available in both Microsoft Forms and Qualtrics and should be used with care as the branching logic needs to be well thought out and implemented correctly for it to be an effective tool.
Finally, determine the best survey tool for your purposes.
The chart below provides some high level information regarding each of the tools available to aid in your decision.
Microsoft Forms
Qualtrics
When to Use
Great for internal or simple and short external surveys.
ex: Survey to be distributed to NYSERDA staff
ex: Simple poll to assess external partner meeting availability
Great for complex external surveys.
ex: Online accessible survey link posted for public engagement
ex: Survey requesting external contacts to respond with requested file attachments
Ease of Use
Easy to use application with simple, essential controls that allows for quick and agile survey creation and communication.
A powerful online survey tool with robust functionality that may require training or guidance to learn how to use.
How to Access
Available to all NYSERDA staff with an active Office365 license.
Any NYSERDA staff member can request a Qualtrics account by email Tricia Gonzales at patricia.gonzales@nyserda.ny.gov.
After logging into the above link, you can access the training entitled Quick Start with Qualtrics that provides a good overview of the application functionality.
Both tools offer the following additional functionality:
Collaboration with colleagues and stakeholders during survey development.
Data export to Microsoft Excel for analysis (this also includes automatic updates for Excel exports when new responses are submitted:
Qualtrics supports real-time updates.
Microsoft Forms supports batched updates requiring users to access the Microsoft Excel export and clicking the Refresh All option in the Data menu of the workbook.
Can be used to create polls or surveys
Create the survey introduction.
You survey introduction should include/consider the following:
Brief description of the survey effort, goals, objectives and/or long-term outcomes.
How the survey benefits the respondent (i.e., why is it important that they complete this survey).
Estimated amount of time it will take a participant to complete the survey.
How and where the data will be used.
If the survey responses will be kept anonymous.
How long participants have to respond to the survey before the response period ends.
Contact name, email and phone number of NYSERDA staff at the top of the introduction to help participants verify the legitimacy of the survey.
Contact name, email and phone number at the bottom of the introduction for participants if they require technical assistance or if they have questions (can be the same contact as above).
Disclaimer
NYSERDA is not able to promise complete confidentiality of survey answers because we are a Public Authority subject to FOIL. If your survey responses should be treated as confidential, please include the following language: The information you provide will be kept confidential to the extent permitted by law including, but not limited to, the Freedom of Information Law (FOIL).
Document your questions in the survey tool.
For each question in a survey, consider how well it addresses what you are trying to measure. Content-related questions you can use to determine the efficacy of your survey questions are:
Is the question absolutely necessary to ensure you obtain useful information?
Are several questions needed or will one suffice instead?
Do participants have all the information needed to accurately and complete answer the questions?
Is the question specific or general enough?
Does the question exhibit a tendency to be biased or loaded?
Is the question worded in a way that will prevent the participant from answering truthfully?
Evaluate the answer choices, placement and rating scale of your questions.
Answer Choices
Possible participant response options should be exhaustive and mutually exclusive.
Question Placement
The first few questions set the tone for the survey and can help put your respondent at ease.
More sensitive or difficult questions should appear later in the survey.
Try to ensure that possible answers to questions are only influenced by previous questions intentionally, such as when deploying branching.
Rearrange questions as needed to encourage a positive experience and garner interest in participants to respond to all questions.
Try to find the flow in the survey so that participants feel encouraged to continue.
Question Rating Scales
1-5 and 1-7 are optimal rating scale ranges.
Always include descriptive labeling for each increment in the rating scale range.
Include a neutral mid-point by employing an odd rating scale range, rather than an even one to prevent bias in the rating scale.
Evaluate the efficacy of your questions.
What to Avoid
Examples
Multiple Subject Questions
Double-barreled Questions are questions that contain two arguments/subjects.
example: Are you satisfied or dissatisfied with the audit program sign-up process and the time it took to schedule your audit?
Double-negative Questions are questions that employ two negative words such as no and not.
example: Do you agree that people who save energy is not uncommon?
Response Option Overlaps
If you ask a question such as:
example: How many LED bulbs did you purchase?
Your response options should include response options such as:
0-5
6-10
11-15
16+
Rather than incorrect response options such as:
0-5
5-10
10-15
15+
Leading Questions
Questions that influence the way in which respondents answer your questions.
example: You like NYSERDA’s new logo, don’t you?
Loaded Questions
Questions that suggest a socially desirable answer or can be emotionally charged.
example: Is it terrible for people to waste limited resources?
Assumptions
Avoid questions that assume a respondents familiarity with specific subjects.
example: Does a lack of courtesy shown by hotel receptionists, in your opinion, influence the customer experience of at hotels?
Jargon or Slang
Use words that are direct and familiar to the respondents.
Social Desirability Bias
People naturally want to appear as good, upstanding members of their community to others will sometimes distort their survey response to encourage that appearance. Survey questions about behaviors or attitudes that are socially-desirable or undesirable can suffer from response bias.
Wording that may encourage Social Desirability Bias:
example: I’m will read a list of actions you may have taken to reduce the amount of energy your household uses. After reading each one, please tell me if this is something you have done.
Instead, try:
example: In talking with people about ways they might reduce the amount of energy their household uses, we often find that people aren’t sure what actions will save them energy or they are too busy make changes in their routines. I’m will read a list of actions you may have taken to reduce energy consumption. After reading each one, please tell me if this is something you have done.
Self-administered surveys, such as web-based questionnaires, are less likely to suffer from social desirability bias than a telephone or in-person survey.
Interviewing Techniques
Good interview techniques include:
Using the interview questionnaire informally, but carefully.
Ask particularly relevant questions verbatim to standardized across respondents.
Follow the question order to properly build a foundation for later questions.
Ask every question for every respondent unless a question is deemed not applicable.
Avoid finishing respondent sentences.
Allow time for respondents to answer and ask additional probing questions if necessary.
Factual Questions and Recall Error:
Ask longer questions (not necessarily more complicated ones).
Introduce questions by identifying the question's subject.
Encourage respondents to take their time.
Stress the importance of collecting accurate information.
Select reasonable recall time-frames.
Shorter time frames should be utilized for less important subjects.
Time frames that are too short can encourage respondents to report behaviors that happened earlier.
Use aided-recall techniques to prompt the respondent’s memory.
Pre-Testing
Once a survey is designed, pre-testing is necessary.
Before conducting a survey of your target audience, test it on a small number of people.
After they have completed the survey, solicit feedback to help uncover issues.
If no significant changes or issues with the survey emerge, the pre-test data may be used as part of the survey analysis.
