Running Studies with Course-level Online Credit Limits

If you haven’t configured your Sona site to set Online Credit Limits on your Course Listings page (and don’t plan to anytime soon), this guide won’t be of much use to you- yet. You can learn more about this feature and how to enable it using our overview and introduction to this feature here: Setting Online Credit Limits on a Course-by-Course Basis.

The point is, we will be covering how online credit limits set on a course-by-course basis (on your Course Listings page) work with study participation and credit assignment. This is only one of several options Sona offers for handling Online Credit Limits, but if you haven’t yet heard of it, it is definitely worth checking out the link above! You will, however, want to do this before continuing, as after this sentence finishes, we will be making the assumption that you are looking for information how your system enforces the course credit limits you set.

More specifically, whether you’ve already set certain Online Credit Limits for a particular course or courses in your Sona site, or you are considering doing so, it is only natural to wonder what effects setting a limit may have for participants’ study participation, the credit granting process, and other ways that a course-level limit may have broader impacts. That’s an excellent question, and one that we took a great deal of time considering during development. For us, the question was how to ensure the only broader impacts you had to deal with were positive. This Online Credit Limit option (i.e., the option to set the Online Credit Limits in your System Settings to a per-course basis), was our answer. This post, on the other hand, is about giving you the answers to how it works. More specifically, how it works in terms of adding studies, study sign-ups, credit assignment, and even to tackle the more complicated type of relevant study: Multi-part online studies.

For standard studies, and even for most online studies, very little has changed that isn’t already covered in our post (linked to above) introducing the new course-level Online Credit Limit option. In a way, you could think of this material as additional information directly related to the study participation and credit compensation process. We’ll begin with the simplest case.

Dealing as they do with limits to the number of online credits that participants can assign to a particular course, such limits have little affect on participant sign-up options for in-person studies (either standard or multi-part). Rather, the course-specific Online Credit Limit begins to make a difference when participants complete online studies. With the exception of online, multi-part studies, the change to the study sign-up and credit granting process is straightforward. If a participant is in one or more courses with Online Credit Limits, then they will not be able to exceed the Online Credit Limits for these courses. These limits are enforced by reducing the number of courses a participant sees listed when assigning credit earned from online study participation:

If they have reached the Online Credit Limit for a course, or the credit earned from online study participation would exceed the amount remaining before reaching this limit is reached, then then the course will not be among those listed for credit assignment. The same is true for credit reassignment.

An example may help make this more readily apparent. To provide you with an example, we’ll need a participant, an online study, and some courses with online credit limits. Throughout this guide, we’ll be relying on a participant who has kindly agreed to help illustrate various scenarios. So we should probably start by introducing you to Tilottama, who prefers to be called “Tilo” for short:

Tilo is currently enrolled in the following courses, shown below along with online credit earned towards each:

Note that Tilo has already reached the allowed number of online credits for the first course. For the second, she’s already assigned one out of the two possible online credits. Finally, Tilo has yet to assign any online credits to her third course, which has an online credit limit of 2.

It so happens that one of the studies Tilo can select after logging into her participant account is an online study she’s interested in. The online study is worth 2 credits, so it benefits everybody as we will be able to use this scenario as our first, concrete example.
During the sign-up process, Tilo has to pick a course to assign the 2 credits she will earn for participation. If you want, you can test yourself here by answering the following question:

Question: Which of these courses will Tilo see (that is, which course(s) will she be able to assign the credit to)?

Answer: We can answer by running down the list and seeing which courses have online credits “remaining”, which here means the difference between the maximum set by the Online Credit Limit and the amount earned or assigned so far. The first course will not be listed because the limit is already reached. The second course has only one online credit available, so the 2 earned would exceed the limit. It too will not be listed. This leaves Tilo only one option. The answer, then is that the only course Tilo will see is her Statistics for the Managerial Sciences course (which has 2 online credits “remaining”, because Online Credit Limit – amount earned/assigned is 2-0=2).

It’s important to note (both for Tilo and for us) that, having assigned these two online credits to her Statistics for the Managerial Sciences course, Tilo has reached this course’s Online Credit Limit. Of the two credits required for her Analytics and Informatics course, one can come from participation in an online study. After she assigns the remaining credit to this course, however, Tilo will no longer be able to assign online credit to any of the three courses from the table above. This means Tilo won’t be able to fulfill the credit requirements (if any exist) for these courses by participation in online studies.

All of the above is the same for participant credit reassignment (if enabled). The one important difference is that administrators can assign online credit a participant has earned to a course with an Online Credit Limit, even if this would go over the maximum allowed. However, to ensure that administrators doing this are aware they are exceeding the online credit limits, they will see a warning when attempt to:

That’s really all there is for study sign-ups and credit assignment with the one exception: Multi-part, online studies.

It’s time to turn to the one study type that is somewhat more involved. The reason for the additional considerations has to do with the way credit assignment works specifically (and uniquely) for multi-part online studies. When participants sign-up for multi-part studies, they are in effect committing to participation in 2, 3, or 4 separate studies, but all at the same time. When this is done for an online study, it means that the credit assignment for each part of the study has to be considered all at once. Luckily, most of the “consideration” is done automatically, by the system. Thus, you only have to be aware of what the system takes into account for studies of this type, so that you can shift any extra burden to the system (a load we’ve developed it to bear) rather than you and/or your researchers (who may be the ones posting such studies on your Sona site).

The basic operation is the same as in the case for standard, one-part online studies. It’s just that, in the case of online multi-part studies, the system will not allow participants to proceed if the total online credits can combine in a manner that will allow the participants to exceed the online credit limits for their courses.

The system, however, does not know what courses your participants might want to assign courses to in advance (we had initially included a precognitively calibrated telepathic interface, but removed it when we realized the ethical implications of such mind-reading capacity). Participants must first select a course for each part of the Multi-Part, Online Study that they wish to assign the online credits they will earn. For example, if the study is a three-part online study, then the participant will select three courses from a dropdown menu. After the participants have selected the courses they wish, the system will determine if the desired combination exceeds the Online Credit Limit for any of the courses.

To make this clearer, we’ll ask Tilo to help out once again, now using the new courses she’s taking for this section. We’ll start there, with Tilo’s list of this section’s courses and their respective limits:

With Tilo back, and with some courses to use for our examples, we’ll start with the simplest possible case: A two-part online study.

We do need one more element for our scenario here: the actual study. This time, the name of the study Tilo has opted into is The CH Social Inequality Study. This study has two parts, each run by a very distinguished researcher, and in deference to them we’ll refer to these parts as the Clauser Part (part 1) and the Horne Part (part 2):

Sign-up Process: The Clauser Part

Now we’re ready to see what happens when Tilo begins the sign-up process. Remember, she’ll have to select two courses (one for each part of the CH Inequality study) to assign the credits to. Tilo has been kind enough to take us through the decision making progress she used, and we’ll cover some alternative scenarios that she might have used, but keep in mind that we’re taking this step-by-step to illustrate not only what participants will do, but also how your system implements the course-level Online Credit Limits you’ve set on your Course Listings page. During and actual sign-up for a Multi-Part, Online Study, these two processes happen one after the other, not concurrently. Participants would select the courses they’d like to assign credit to for each part of the study, and then the system would implement the procedure we’ll be going over below, starting with Tilo’s initial selection:

The green arrows indicate that Tilo can assign the credit from the Clauser Part of the CH Social Inequality Study to any of here three courses. Let’s say she picks Research Topics in HCI. What happens? Well, nothing yet because she has to finish assigning courses so that the system can ensure the selection is allowable, but Tilo is certainly capable of thinking the process through. From her perspective, now that she’s made a choice for the Clauser Part, this changes how she should think about assigning the credit she will earn from the Horne Part part of this two-part, online study.

The Horne Part

This is where Tilo’s help will be particularly useful, as she can take us through her thought process during her choice of course selections for each of the CH Inequality study’s two parts. That way, we can consider how the system would implement her choice (and other choices she might have made) while going over how the system would allow or not allow a particular combination of choices:

As indicated by the now red arrows (as well as the crossed-out courses), two of the possible options Tilo had for assigning credit to her courses for the Clauser Part are, from her perspective, no longer “available”. She’s crossed off the Analytics course because the Horne Part of the CH Social Inequality study exceeds the Online Credit Limit for this course. The reason she won’t assign these 2 credits to her Research Topics in HCI, however, is because of her choice of assignment for the Clauser part of the study. The only remaining option of the three is the Academic Culture & Research Practices Seminar.

Suppose Tilo had initially assigned the 1 credit for the Clauser Part to her Analytics for Educational Data Science course. Then, when selecting a course for the Horne Part of the study, she could consider this as crossed off (it has an Online Credit Limit of 1). However, because her earned/assigned online credits towards the Research Topics in HCI course is 0, and she hasn’t mentally counted any credits towards it as she did earlier, she can still has two “remaining” credits she can use. This is, in fact, another combination she considered, and it is one that the system would have (after she’d selected them) permitted, allowing her to proceed with the study sign-up.

Variations for Various Choices

It’s easy to imagine how this same scenario might play out for different values for the three credit limits, the credits already counted towards these limits at the start of this two-part study, and the number of credits each study part was worth. Suppose, for example, that both the Clauser and Horne parts were worth 2 credits. Then Tilo would have ruled out her Analytics for Educational Data Science course listed as a credit assignment option at the start.

As another variant of this two-part study scenario, what if both the Clauser and Horne parts of the study were worth 3 credits each? Looking at the 0’s in the earned/assigned credit column of her course table, it’s clear that the remaining available online credit Tilo can assign is 6, and last we checked, 3+3=6. But (and this is key!), just because the total remaining matches the total earned for an online, multi-part study, this doesn’t ensure that the participant can proceed with the sign-up. Tilo wouldn’t have been able to, because no matter how she selected her two courses from the three she’s considering, there’s no way to do so that won’t end up exceeding one of the couse’s Online Credit Limits.

Concluding Example 1

And that concludes our two-part, online study example. We have just one more to go, but luckily now that we’ve waded into the somewhat murkier (but nonetheless beautiful) waters of Multi-Part, Online Studies, we’ve covered more than you might think. In our final example, we hope to show you that you may know more than you think. At the very least, we want to demonstrate that once you understand how the two-part, online study works, adding more parts doesn’t make things more conceptually difficult or complex. To do this, we’ll jump right to the maximum number of parts and run through a four-part, online study.

So as to not start from scratch after doing all this work together, we’re going to treat the four-part study by extending the previous example to one of four parts. In other words, we will use the same courses from Example 1 with the same values, but we’ll imagine that this time Tilo sees a similar looking study to that from the previous example. Two of the researchers running the studies are, indeed, the same! This time, however, the study is a four-part, online study called the CHSH Socioeconomic Inequality study. And, while Clauser and Horne are still eminent researchers involved in the study, they are joined by the distinguished scientists Shimony and Holt. As before, we will name each study part after one of these, keeping the Clauser Part and the Horne Part the same. Now, though we have two new parts:

Although we’re starting as if Tilo is in the middle of the sign-up process (about to assign 2 credits to a course for the Shimony Part of this study), this time Tilo’s choices won’t be the same. Previously, she assigned 1 credit to Research Topics in HCI and then 2 credits to her Academic Culture & Research Practices Seminar, and then the system determined the combination did not exceed any course-level Online Credit Limits, and let her proceed with the study sign-up.

If Tilo makes the same two course choices for the CHSH Socioeconomic study, though, she knows she’ll end up with a combination that isn’t allowed: Any combination she’d end up with would mean exceeding an Online Credit Limit for at least one of her courses because her first two choices meant leaving no course with two “remaining” credits.

Knowing this, Tilo was more careful during the sign-up process for the CHSH Socioeconomic Inequality study. She knew to consider credit assignment a bit more carefully, and chose to assign the 1 credit from the Clauser Part to her Analytics for Educational Data Science course (though opted to make the same choice for the 2 credits from the Horne Part). This, then, is how her credit situation stands during the sign-up process as she is tasked with assigning credits to courses for the Shimony Part of this four-part, online study:

The Shimony Part We’re going to mix things up a bit, and have Tilo wait while you consider how she might proceed with the next two steps. You know from the tables above which courses have how many “remaining” credits, and that the Shimony Part is worth 2 credits. How might you proceed?

You might start with filling in the boxes (mentally, anyway, as Tilo did) with the appropriate number of “remaining” credits, and then the answer becomes quite simple. Only one course has enough remaining credits for Tilo to assign them to: Research Topics in HCI. So that’s where she assigns the 2 credits from the Shimony Part of the CSHS Socioeconomic Inequality study.

The Holt Part

Before we look at the final part of Tilo’s study sign-up (namely, the Holt Part of the study), let’s quickly look at her updated credit situation:

The table immediately above, together with the table displaying the credit value for each part of the CHSH Socioeconomic Inequality study, makes this last part a breeze. Tilo’s current credit calculations (say that 10 times fast!) tells her she has only one course option with any “remaining” online credits. But that’s not a problem, because the Holt Part of this study is worth 1 credit. Tilo selection combination won’t cause any limits to be exceeded if she assigns the 1 credit for the Holt Part to her Analytics for Educational Data Science course. She does, the system agrees that the combination doesn’t exceed any Online Credit Limits, and she and we are now good to go!

We hope that Example 2 did, indeed, convince you that once you understand how course-level Online Credit Limits work for a two-part, online study, you really understand how they work for Multi-Part, Online Studies in general. In fact, if you go back and look over how this process works even in the case of a standard, online study, you may see that, despite the seeming complexities, the process is actually straightforward. That’s because we took care of all the would-be complexities during the development of the per-Course Online Credit Limit option.

The difference between a two-part study and a three- or four-part study is in the number of ways study parts can combine during the sign-up process so that the participant won’t be able to proceed. This goes back to the main difference between Multi-Part, Online Studies in general and the standard variety. In the former case, it’s not simply about having enough “remaining credits” (as it is in the latter). For the sign-up to proceed, the participant must have enough “remaining credit” in at least one course for each part of the Multi-Part Online Study. Put simply, just because the total number of remaining online credits in a participants’ courses matches the total number of credits for an Online, Multi-Part Study doesn’t guarantee a successful sign-up process.

Success here comes through a bit of proper prior planning. That’s why we’re very thankful to Tilo for helping us demonstrate the kind of consideration that goes into signing up for online studies, whether they are multi-part or no. We hope that her examples here help you to better understand how the per-course Online Credit Limit option works with studies in your Sona site.