Qualtrics is a very powerful online survey generator service that Bates now has a college-wide license for. In addition to being a great utility, they have great online documentation. The purpose of this website is to provide an easy reference for commonly used survey designs that I've seen our psychology majors use. If you have other questions, I encourage you to check out:
- Bates ILS page about Qualtrics
- Qualtrics online help system, including a detailed knowledge base and how-to videos
- The Qualtrics utility itself, which has lots of documentation and tips built in. (Note: Bates login required.)
This page is part of the Technical Resources for Students section of the Bates College Psychology Department website.
Follow the links below to learn about different survey configurations, or scroll down the page to read them all!
Note: Qualtrics redesigned their interface after some of these screenshots were taken. The functionality is largely the same, but your screen may look slightly different from these screenshots as a result of the redesign.
How do I.....
Counterbalance to prevent ordering effects
Randomly assign conditions (between-subjects)
Update "confirmation" text (the last page your participants see AFTER submitting data)
Change survey background/colors
Alter questions based on a previous response
Time participants on my survey
Distribute my completed survey
Include a raffle or participation credit at the end of my survey
Counterbalance to prevent ordering effects
The goal of counter-balancing is to present different sections of your survey in a different order to different participants. The key to counterbalancing is to create different blocks of questions representing the sections that should be reordered.
Once you have set up your blocks, you can then configure your Survey Flow to present these blocks in a different order for different participants. In this simple example, I have two blocks "Block1" and "Block2". I want all participants to answer questions in both blocks, but some of them should take one first and some the other first. I also start off with a block called "Consent" which should always appear first.
Once your blocks are defined (you don't need to have all of your questions entered yet), go to Survey Flow and set up your survey like this:
I start my Consent block and then insert a Randomizer object which has the other two blocks below it.
There are two important settings on the Randomizer object. The first is "Randomly present 2 of the following elements" which is the default setting. This means that each participant will be presented with both blocks. The second is the option to "Evenly Present Elements". If checked, an attempt will be made to assign participants equally across ordering conditions. If unchecked, the assignment will be truly random and a balanced N is not guaranteed.
Randomly Assign Conditions
One of the most common applications of random assignment is to set up a survey that is identical in most regards, but with one element that varies among participants. For example, participants might be shown different images as primes or be asked to read different text passages. In order to do this in Qualtrics, you can use a simple Randomizer object in the Survey Flow along with the Display Logic setting for the object that is to be randomized.
In this example, I want to randomly assign participants to one of two different sets of instructions. Otherwise the surveys will be identical. First, go to Survey Flow and create a Randomizer object. Use the Embedded Data object beneath the Randomizer to set a variable to one of two values. In this example, my variable is called "text_condition".
With that configured, I can define the two different sets of instructions as two different objects in the appropriate place on my survey. In order to prevent both from being displayed, click on the gear icon to the left of each object and choose "Add Display Logic". Now I can enter my variable (text_condition) and what value is necessary for this object to be displayed.
There are two important settings on the Randomizer object. The first is "Randomly present 1 of the following elements". This means that each participant will only receive one order instead of being assigned to repeat the same blocks of questions multiple times. The second is the option to "Evenly Present Elements". If checked, an attempt will be made to assign particpants equally across conditions. If unchecked, the assignment will be truly random and a balanced N is not guaranteed.
Update "confirmation" text (the last page your participants see AFTER submitting data)
At the end of your survey, there is an End-of-Survey message with the default text "We thank you for your time spent taking this survey. Your response has been recorded." To change this text, click on the End-of-Survey message. In the left configuration pane, change the dropbox from Default to Custom. This will enable a second dropbox labeled "Load a Saved Message". Navigate to My library: New Message. (see image below)
Once you have changed the text, be sure to fill in description field with something (like "debriefing page") or you will get an error message "Description is not valid." (The description won't be displayed to your participants.)
To edit this text, go to the Library link hidden beneath the 3 line icon in the upper
left corner of the
qualtrics page. (See screenshot below.)
Then choose the Messages Library tab. To return to editing your survey, click on the Projects link in the upper right corner.
Note: Editing this confirmation text is an essential step if you are offering your online
participants credit from the psychology participation pool. You will
need to update the confirmation text to include your experiment number,
codeword, and a link to the participation form. (When pasting the link, make sure it appears as a blue link. If it doesn't, highlight that line separately, click on the link icon
and paste the web address again in the link dialog.)
Change survey background/colors
While editing your survey, click on the "Look and Feel" icon at the top left. In the bottom left corner, click on "Back to Old Editor"
This will show you the default Bates template, with a photo of Mt. David as the background and Bates colors for the different page elements. From here you can edit different components of the survey's appearance. However, the most efficient way to alter the look of your survey is to use one of the predefined Qualtrics templates.
The interface is a little non-intuitive. The box on the left has two dropboxes which both say "Bates College - Site Wide" by default. Click on the top drop box to select Qualtrics templates:
And then click on the bottom drop box (which still says "Bates College - Site Wide" for some reason) to reveal a variety of templates. My favorite for a straight-forward, clean look is "Minimal". If you like, you can use this as a starting point and add additional color options to it using the Look and Feel interface.
Alter questions based on a previous response
There are two ways to alter the text of a question based on a response to a previous question. We'll start with the simple approach.
The Simple Approach
You can use this method if you simply want the answer from a previous question to appear in the text of a subsequent question. For example, I can ask people what major they have declared and then use that information to ask about their major.
First create a question asking for their major. Then create a subsequent question on another page, either in a different block or separated by a page break. As you are editing this question text, put your cursor where you want their answer to be inserted, and click on "piped text". Drill down in this menu to select "Pipe text from a .... survey question". Choose the source question and then click "Selected Choices".
After you have done this, Qualtrics will insert code into your text that looks like this:
You can copy and paste this code in your text in case you want their answer to appear more than once, or if you want to move it. Be sure to copy the enter code, though!
Tip:I recommend making your source question ("what is your major?" in this example) a required question otherwise your subsequent questions that rely on this answer will have blank spots if the participant doesn't answer the question.
But what if you want something more complicated than just the text of an answer to appear in subsequent questions? That brings up to the complicated approach.
The Complicated Approach
For this example, let's imagine that I want to change the text of some questions based on the response to a previous question, but that the text substitutions need to be more complicated than just plugging the answer in verbatim. For example, if I ask people what gender they would prefer to date, and then I presented a scenario customized for that gender. I would want the text to say "You see a man in a cafe. He is reading a newspaper and sipping his coffee." if the participant indicated "male". If the participant indicated "female", I want to change the text to "You see a woman in a cafe. She is reading a newspaper and sipping her coffee."
I can use a combination of the Survey Flow (branch and embedded data) and piped text to accomplish this result.
1. Set up a source question asking for the preferred gender that your participant would like to date. Answers are "Male" or "Female".
2. Open up the Survey Flow and create a Branch object. This branch object should be placed sometime after the question block which contains your source question but before the block(s) containing your customized questions.
3. Set the condition for your Branch object to reference your source question, and select one of the possible answers ("Male" for example.)
4. Underneath the branch, created embedded data objects for each of the words that will vary depending on the participant's response. In our example, we want to change "man/woman", "He/She", and "his/her" so we will need three variables (embedded data objects).
5. Once this is completed, duplicate the entire branch and edit it to change the response to "Female". Also edit the embedded data object values to match this other choice. The final result will look like this:
6. Now we can reference this embedded data objects using piped text similar to how we did it in the simple approach. As you are editing the target question text, put your cursor where you want the custom text to be inserted, and click on "piped text". Drill down in this menu to select "Pipe text from a... Embedded Data Field" In the box provided, type the appropriate variable name that you defined in your survey flow.
Here is the customized question with all three data fields referenced:
Tip:I recommend making your source question ("what gender do you prefer to date" in this example) a required question otherwise your subsequent questions that rely on this answer will have blank spots if the participant doesn't answer the question.
Play a video in my survey
There are two ways to play a video in your Qualtrics survey, either by
uploading a video file to your library, or by linking to a video that is
already online.
Qualtrics help covers this topic in great detail, so I won't duplicate
their effort. I will offer my recommendation to use the embedded
youtube video
option if you can. This is an easy approach for getting around potential
browser compatibility issues with a video file uploaded directly to
Qualtrics. At the end of their help page on this topic, they offer
some tips on configuring youtube so that your video can be private even
though it is on youtube.
Read the
Qualtrics video help page here.
For my recommended method, use the instructions under "Adding
media using HTML"
Time participants on my survey
The "Timing" question type can be very useful for keeping track of how long participants spend on a given page. It has configuration options that allow you to show or hide the timer from your participants. However, the Timing object is limited to a single page of your survey. What if you want to time the entire survey instead of a single page? Qualtrics has a hidden feature to do just that. You can just add an Embedded Data element to your survey flow named "Q_TotalDuration". This field will report in the results the total number of seconds the participant spent on the survey.
Distribute my completed survey
Once you are ready to collect data, click on the Distributions tab at the top of your survey. Qualtrics offers many ways to distribute your survey to potential participants. For most purposes in research psychology, the best one is to get a single, anonymous link that you can send out to multiple people. To do this click on the green button "Get a single reusable link". Once you have clicked this, a link will appear which you can send to potential participants via email, facebook, etc.
Export data to SPSS
Once you have collected some data, you can export the results in various formats, including into an SPSS .sav file. To do this, click on the "Data & Analysis" tab at the top of your survey. On the right side of your browser, there is a button labeled "Export & Import". Click this button and choose "Export Data".
- At the top of the dialog, choose SPSS.
- In the bottom left, click "More Options"
- Scroll down and uncheck the box labeled "Compress data as .zip file"
- If you have any sort of randomization or counter-balancing in your survey, you can check the checkbox under "Export viewing order data for randomized surveys" to include a variable in SPSS that will show which randomized conditions each participant was assigned to.
- Finally, click the green Download button.
A file notification will appear in your browser when the file download is complete. You can click on this notification to open the SPSS file.
Include a raffle or participation credit at the end of my survey
Credit: Participation credit for online studies in the Bates psychology department is managed through a separate survey which your participants will need to fill out after they have completed your study. Once your study is approved (you can apply here), you will be given an experiment number, a code word, and a link to the department's survey. You need to edit your debriefing page to include this information at the end of your survey.
Raffle: The same approach can be done if you wish to have your participants submit their contact info into a raffle for a chance to win money or some other compensation. If you create a separate survey to collect that information, you can link to that survey from your debriefing page.
Both: The trick comes in when you wish to offer both types of compensation. It is against department policy for students to get credit and money (or a chance at money) for the same study. For this reason, you need to have branching logic in your survey flow which directs your participants to one of two debriefing pages. Here's how to do that:
1. Follow these instructions to set up a custom end-of-survey message. Let's assume that this debriefing page will be for your participants who want to enter the raffle.
2. In your messages library, set up a second end-of-survey message with different text designed for your credit participants. Since your participants will only see one of these messages, be sure that any information that applies to all participants is contained in both messages.
3. At the end of your survey, set up a multiple choice question which asks which type of compensation they prefer.
4. In the survey flow at the very end, configure a branch object with the condition based on the response to the question. Since our first end-of-survey message will lead to the raffle page, we need to divert people who choose credit instead.
5. Below that branch object, create an End of Survey object. Click on "Customize", check the box to "override survey options", and choose "Custom end of survey message..." Here you will link to your second debriefing page with the credit information.
Since anyone who doesn't choose Course Credit for that question will just go to the first debriefing page you set up, there is no need to set up a second branch/end-of-survey object in your survey flow for the raffle condition.
This page is part of the Technical Resources for Students section of the Bates College Psychology Department website.