Versions Compared

Old Version 1

changes.mady.by.user Andy R. Andersen

Saved on

compared with

New Version Current

changes.mady.by.user Andy R. Andersen

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Aura - Title
color#0073a4
textAlignleft
fontSize28
lineHeight28
cloudTextDemo Title
tagh1
fontWeightbold
Forms: Creating an Effective Survey

Aura - Divider
serializedStyles{"alignment":{"horizontal":"start"},"icon":{"name":"faPaperPlane","color":"#333","size":24},"text":{"color":"#333","fontSize":14,"textAlign":"left","fontWeight":"bold","text":"Aura Divider"},"border":{"top":false,"right":false,"bottom":true,"left":false,"color":"#0073a4","style":"solid","width":2},"size":{}}
typeregular

Info
iconfalse

This page provides information and details regarding the two preferred survey tools at NYSERDA: Microsoft Forms and Qualtrics, along with other important information you should know before attempting to create a survey with either of these tools.

  • Consult your evaluation liaison if you need help to develop a complex survey or if you have questions regarding survey design.
  • Public facing surveys should always be reviewed by the Marketing team and your Program Attorney before being deployed.
    (The Marketing review focuses on evaluating comprehension and customer experience and will engage the Communication team as needed.)

Process Overview

The below steps outline the high-level process when creating a new survey. Click each tab to learn more about the actions that should be taken in each major step of the process. Should you have any questions about the information on this page, or when developing a survey, consult with your evaluation liaison.

Aura - Tab Group
settings{"generalSettings":{"tabSpacing":0,"tabWidth":100,"tabHeight":40,"direction":"horizontal"},"activeSettings":{"backgroundColor":{"color":"#002d72"},"text":{"fontSize":16,"color":"#ffffff","textAlign":"left","fontWeight":"normal"}},"inactiveSettings":{"backgroundColor":{"color":"#ffffff"},"text":{"fontSize":16,"color":"#5e6c84","textAlign":"left","fontWeight":"normal"}},"contentSettings":{"backgroundColor":{"color":"#fff"},"padding":{"top":10,"right":10,"bottom":10,"left":10},"border":{"style":"solid","width":2,"top":true,"bottom":false,"left":false,"right":false,"color":"#002d72"}},"hoverSettings":{"backgroundColor":{"color":"#0073a4"},"text":{"fontSize":16,"color":"#ffffff","textAlign":"left","fontWeight":"normal"}}}
tab1
Aura - Tab
title1: Prepping the Survey
UI Steps
sizesmall
UI Step

Outline the requirements for your survey.

  • What is the purpose or goal of the survey?
  • What data/information are you looking to capture and what will it be used for?
  • Who will be responding to the survey?
  • Are the potential respondents internal or external stakeholders?
  • Are there other colleagues at NYSERDA who could benefit from the data collected? If yes, include them in the survey development.
UI Step

Determine the Type of Survey that will best meet your needs.

Survey TypeDescription
Questionnaires

Are usually completed independently by respondents. Questionnaires are more likely to return Quantitative data. Quantitative data can be counted, measured, and expressed either categorically or numerically.

example: communicated poll utilizing a rating scale or closed-ended questions (yes/no) to determine who can attend an upcoming meeting.

Interviews

Are generally conducted by an interviewer to document participant responses. Interviews are more likely to return Qualitative data. Qualitative data is descriptive, conceptual and can be categorized based on traits or characteristics.

example: research conducted using open-ended questions in a focus group environment to determine user experience efficacy for a new application.


UI Step

Evaluate how long the survey should be. 

  • There is a delicate balance between asking enough questions to gather all needed data while minimizing how long a participant spends responding to a survey. Too few questions and you might not obtain the insights you're looking for and too many questions can alienate the participants, leading to incomplete surveys and less reliable data.
  • In general, a Questionnaire utilizing mostly close-ended questions should take no more than 20 minutes for a participant to complete. Interviews can take longer based on the nature of the questions and the answers provided.
  • There are many factors that can help determine how long a respondent will be engaged in a survey, such as:

    • The survey method used.

    • The type of questions you'll be asking (open versus closed-ended).

    • Possible incentives offered, either real or perceived.

    • The frequency of survey activity.

  • Determining how long you'd want a participant to spend on the survey can help outline what types of questions you need to ask and how they should be worded to avoid confusion.

**Note: While there is no perfect survey length, it is more important to estimate the desired participant response time first, and then generate the required questions according to that scale, rather than the other way around.

UI Step

Outline the Survey Questions. 

Question Categories:
Question CategoryDescription and ExamplesProsCons
Open Ended Questions

Allow respondents to answer the questions in their own words and usually provide a text box for their unique input.

example: What do you think about the program?

answer: 

test test test test test testtest test test test test test


  • Easier to write
  • Allows for more detail
  • May produce unexpected answers
  • More salient
  • More difficult to answer (greater burden on respondent)
  • Must code answers (interpret & categorize responses)
  • Harder to analyze
Close Ended Questions

Require respondents to select their response from a provided list of options

example:  Where  did you hear about the program?

answer options:

  • Website
  • Utility Bill
  • Tv or Radio Advertisement
  • Friends or Family
  • Less expansive
  • Easier to analyze
  • Can test specific responses
  • Harder to write
  • Need a complete list of responses
  • Limited number of responses
Question Types (options in the below list are available in both tools):
Question TypeDescription

Forced Choice

Multiple options to choose from where only one can be selected.
Select-All ChoiceMultiple options to choose from where all options that apply can be selected
DateProvides ability to enter in or select a date.
LikertGenerally a 5 or 7 point scale which allows a respondent to express how much they agree or disagree with a particular statement.
RankingAsks respondents to compare items to each other by placing them in order of preference.
RatingDisplays a scale of answer options from any range (0 to 100, 1 to 10, etc.) where respondents select the number that most accurately represents their response.
TextAllows respondents to enter in a single line or multiple lines of text.

**Note: The above table lists all available questions types that are available with Microsoft Forms. Qualtrics, however, has over 100 possible questions type, most of which are not displayed in the table above. 

Branching:

Branching is a more complex survey function that allows a different set of questions to be shown to respondents based on how they answered the previous question. This function is available in both Microsoft Forms and Qualtrics and should be used with care as the branching logic needs to be well thought out and implemented correctly for it to be an effective tool.


UI Step

Finally, determine the best survey tool for your purposes. 

The chart below provides some high level information regarding each of the tools available to aid in your decision.


Microsoft Forms
Qualtrics

When to Use

Great for internal or simple and short external surveys.

  • ex: Survey to be distributed to NYSERDA staff

  • ex: Simple poll to assess external partner meeting availability

Great for complex external surveys.

  • ex: Online accessible survey link posted for public engagement

  • ex: Survey requesting external contacts to respond with requested file attachments

Ease of Use

Easy to use application with simple, essential controls that allows for quick and agile survey creation and communication.

A powerful online survey tool with robust functionality that may require training or guidance to learn how to use.

How to Access

Available to all NYSERDA staff with an active Office365 license.

Any NYSERDA staff member can request a Qualtrics account by email Tricia Gonzales at patricia.gonzales@nyserda.ny.gov.

How to Use

Page Tree
rootMicrosoft Forms

Info
iconfalse

After logging into the above link, you can access the training entitled Quick Start with Qualtrics that provides a good overview of the application functionality.

Both tools offer the following additional functionality:

  • Collaboration with colleagues and stakeholders during survey development.
  • Data export to Microsoft Excel for analysis (this also includes automatic updates for Excel exports when new responses are submitted:
    • Qualtrics supports real-time updates.
    • Microsoft Forms supports batched updates requiring users to access the Microsoft Excel export and clicking the Refresh All option in the Data menu of the workbook.
  • Can be used to create polls or surveys




Aura - Tab
title2: Creating the Survey
UI Steps
sizesmall


UI Step

Create the survey introduction. 

You survey introduction should include/consider the following:

  • Brief description of the survey effort, goals, objectives and/or long-term outcomes.
  • How the survey benefits the respondent (i.e., why is it important that they complete this survey).
  • Estimated amount of time it will take a participant to complete the survey.
  • How and where the data will be used.
  • If the survey responses will be kept anonymous.
  • How long participants have to respond to the survey before the response period ends.
  • Contact name, email and phone number of NYSERDA staff at the top of the introduction to help participants verify the legitimacy of the survey.
  • Contact name, email and phone number at the bottom of the introduction for participants if they require technical assistance or if they have questions (can be the same contact as above). 
Warning
titleDisclaimer

NYSERDA is not able to promise complete confidentiality of survey answers because we are a Public Authority subject to FOIL.  If your survey responses should be treated as confidential, please include the following language: The information you provide will be kept confidential to the extent permitted by law including, but not limited to, the Freedom of Information Law (FOIL).


UI Step

Document your questions in the survey tool.

For each question in a survey, consider how well it addresses what you are trying to measure. Content-related questions you can use to determine the efficacy of your survey questions are:

  • Is the question absolutely necessary to ensure you obtain useful information?
  • Are several questions needed or will one suffice instead?
  • Do participants have all the information needed to accurately and complete answer the questions?
  • Is the question specific or general enough?
  • Does the question exhibit a tendency to be biased or loaded?
  • Is the question worded in a way that will prevent the participant from answering truthfully?


UI Step

Evaluate the answer choices, placement and rating scale of your questions.

Answer Choices:
  • Possible participant response options should be exhaustive and mutually exclusive.
Question Placement:
  • The first few questions set the tone for the survey and can help put your respondent at ease.
  • More sensitive or difficult questions should appear later in the survey.
  • Try to ensure that possible answers to questions are only influenced by previous questions intentionally, such as when deploying branching.
  • Rearrange questions as needed to encourage a positive experience and garner interest in participants to respond to all questions.
  • Try to find the flow in the survey so that participants feel encouraged to continue.
Question Rating Scales:
  • 1-5 and 1-7 are optimal rating scale ranges.
  • Always include descriptive labeling for each increment in the rating scale range.
  • Include a neutral mid-point by employing an odd rating scale range, rather than an even one to prevent bias in the rating scale.


UI Step

Evaluate the efficacy of your questions.

What to AvoidExamples
Multiple Subject Questions

Double-barreled Questions are questions that contain two arguments/subjects.

example: Are you satisfied or dissatisfied with the audit program sign-up process and the time it took to schedule your audit?

Double-negative Questions are questions that employ two negative words such as no and not.

example: Do you agree that people who save energy is not uncommon?

Response Option Overlaps
If you ask a question such as:

example: How many LED bulbs did you purchase?

Your response options should include response options such as:

  • 0-5
  • 6-10
  • 11-15
  • 16+

Rather than incorrect response options such as:

  • 0-5
  • 5-10
  • 10-15
  • 15+
Leading Questions

Questions that influence the way in which respondents answer your questions.

example: You like NYSERDA’s new logo, don’t you?

Loaded Questions

Questions that suggest a socially desirable answer or can be emotionally charged.

example: Is it terrible for people to waste limited resources?

Assumptions

Avoid questions that assume a respondents familiarity with specific subjects.

example: Does a lack of courtesy shown by hotel receptionists, in your opinion, influence the customer experience of at hotels?

Jargon or SlangUse words that are direct and familiar to the respondents.
Social Desirability Bias

People naturally want to appear as good, upstanding members of their community to others will sometimes distort their survey response to encourage that appearance. Survey questions about behaviors or attitudes that are socially-desirable or undesirable can suffer from response bias.

Wording that may encourage Social Desirability Bias:

example: I’m will read a list of actions you may have taken to reduce the amount of energy your household uses. After reading each one, please tell me if this is something you have done.

Instead, try:

example: In talking with people about ways they might reduce the amount of energy their household uses, we often find that people aren’t sure what actions will save them energy or they are too busy make changes in their routines. I’m will read a list of actions you may have taken to reduce energy consumption. After reading each one, please tell me if this is something you have done.


Info

Self-administered surveys, such as web-based questionnaires, are less likely to suffer from social desirability bias than a telephone or in-person survey.





Aura - Tab
title3: Obtaining Approval
UI Steps
sizesmall


UI Step

Interviewing Techniques

Good interview techniques include:
  • Using the interview questionnaire informally, but carefully.
  • Ask particularly relevant questions verbatim to standardized across respondents.
  • Follow the question order to properly build a foundation for later questions.
  • Ask every question for every respondent unless a question is deemed not applicable.
  • Avoid finishing respondent sentences.
  • Allow time for respondents to answer and ask additional probing questions if necessary.
Factual Questions and Recall Error:
  • Ask longer questions (not necessarily more complicated ones).
  • Introduce questions by identifying the question's subject.
  • Encourage respondents to take their time.
  • Stress the importance of collecting accurate information.
  • Select reasonable recall time-frames.
    • Shorter time frames should be utilized for less important subjects.
    • Time frames that are too short can encourage respondents to report behaviors that happened earlier.
  • Use aided-recall techniques to prompt the respondent’s memory.


UI Step

Pre-Testing

Once a survey is designed, pre-testing is necessary.

  • Before conducting a survey of your target audience, test it on a small number of people.
  • After they have completed the survey, solicit feedback to help uncover issues.
  • If no significant changes or issues with the survey emerge, the pre-test data may be used as part of the survey analysis.




HTML
<style type="text/css">
#title-text { display: none;}
</style>

...