Incentivizing Pre-K Online Gaming With Digital Sticker Books and Pornography (For The Adults Says Heckman, Half Joking)

This is another post with clips culled from talks given at the Center for the Economics of Human Development’s working group, Measuring and Assessing Skills: Real Time Measurement of Cognition, Personality and Behavior. It was held at the University of Chicago in February 2018. I previously shared a segment called from “Math to Marksmanship” with Nobel Prize economist James Heckman, Gregory Chung of UCLA-CRESST and Jeremy Roberts consultant to PBS Kids.

Below are ten additional excerpts from that talk. I watched all two hours and pulled highlights, so you don’t have to.  Topics covered include: game-based learning for pre-schoolers; how to get pre-readers to create online accounts; how digital games can be used to identify “Big Five” behavior traits; and a real doozy, Dr. Heckman’s half-joking suggestion that gamification and incentives of pornography for adults could encourage parents to have their children use online games more often. No, really.

Below is a relationship map of the organizations mentioned in the presentation. You can access an interactive version here.

PBS Kids is a media content provider for children ages 2 to 8 years old. Nearly 65% of all children in that age range interact with PBS Kids’ content at some point during any given month. Their apps have been downloaded 45.5 million times, and they deliver 276 million streams per month across multiple digital platforms. Their key strategy is to try to be wherever the kids already are: desktop, television, mobile, and classroom (whiteboards/chromebooks). They don’t develop their own content, but instead curate properties created by others such as: Daniel Tiger’s Neighborhood (social emotional learning, Mr. Roger’s Neighborhood); Curious George (engineering/math); Pinkalicious (art); and Cat in the Hat (engineering/science).

The US Department of Education recently awarded PBS Kids their fifth consecutive, $100 million, five-year Ready-to-Learn Grant. Each cycle has a different focus. The focus of their current grant is science, informational text, and personalized/adaptive learning. A quarter of the budget is allocated for research carried out in partnership with UCLA CRESST, SRI International, Rockman, and CAST.

PBS Kids is also collaborating on learning analytics with Kidaptive, an AI online learning platform developed in association with Stanford, and with EnLearn, which was created with Gates Foundation funding at the University of Washington Game Science Lab. Games discussed in the presentation included the Measure Up and Super Vision platform, Fish Force, and Kart Kingdom, a starter online multi-player game.

Even though it takes place later in the talk, I want to begin with this excerpt, because it provides a true glimpse into how these people see parents. During another part of the talk Roberts notes that the audience PBS KIDs is focusing on for their grant’s ethnographic study is comprised of Title One parents and those they perceive of as “underserved.” Heckman’s comments are laden with derogatory race/class inferences.

1. Getting Parents to Prioritize the App…Using Pornography Incentives?

Timestamp: 1hour 14 minutes 20 seconds

Clip here

Roberts: The idea here is for us to tease apart any effects that we find that are pursuant to the use of the parent application (Measure Up and Super Vision). Um, yes there’s a lot of debate right now about what you have to do to get parents to want to engage with the tool. Is this just a no-brainer? Are they going to use it? No, it’s a key challenge.

Parents will tell you that they really care, and they will tell you that they want to be a good parent, but it’s aspirational. They will repeatedly give you false information about their expectations for how they will use the product. It isn’t until you pin them down with crafty, crafty strategies before you can start to get the truth.

One example would be the third time…I had them tell me what we wanted to hear through two different trials and finding out what they think about it. Finally, on the third trial the innovation in the design was to have them plot their day BEFORE we show them any products or ask them anything about this, plot their day. Then we had them circle the parts of their day where they think they’re actively involved in parenting. So they did that. Then we showed them all the products, and then we said, hey would you use it? And they told us the same story, yes we would. And then we said now please point on the diagram of your day when you would actually use this, and then they FINALLY they all said, oh yeah, oh, I don’t know, I’m not sure…because they’re so busy.

Other Person: But don’t you think being able to tie it to something that would directly show their school readiness might be able to get parents more involved?

Roberts: I want that to be true. Yeah. Are you asking what I want? (laughter from the table)

Other Person: Is that something, a strategy, that you’ve explored further?

Roberts: Uh, it’s something that we’re looking at now. We’re definitely not giving up on this. This is key. We already know that when the parent IS involved, that the outcomes are way better.

Heckman: (interjects) But what about a “build-a-child” game for the parent? You know? No, in other words you gamify it for the parent. You’re making the parent into some kind of wise educational administrator. They’re not. So you could even have pornography, I guess. Something that would entice them. I’m just, you know. Nah, sorry about pornography, but I’m just thinking how to motivate the parents.

Roberts: We need to innovate here. We need to use some psychology on parents. We need to meet them where they are. We need to use what they have, and we need to let them do…um so, yeah.

2. Gathering Home Life Background Information

Timestamp: 16:05

Clip here

Roberts: We have telemetry on parent use of recommendations associated with the near time kid use of the app.

Heckman: What about the parents’ environment? Some measure of the home in addition to what you see parents are doing with your product.

Roberts: Currently very limited. I would say, right. No we’re not…

Heckman: So you wouldn’t know kind of what effect you might be having?

Roberts: No, but that kind of background information would be amazing. Like that’s the kind of thing…

Heckman: Right, and do you have plans to do that?

Roberts: Not currently, but that doesn’t mean we couldn’t consider it. We have some discretion in terms of what the out years of the grant do in terms of study.

Heckman: Because we see in some of these analyses tremendous changes in family structure, in family approach to the child…

Roberts: Oh sure, poverty, how many words…uh

Heckman: No, exactly, but literally the change in the parental strategies…

Roberts: Change. Yep, no, that’s great. I’d love to talk more. One of the things I would like to get out of this meeting, selfish aspect from my perspective, is I would love to crowd-source ideas about the highest-value study we might permit on datasets like this. Because from my perspective there is more than we could possibly every do based on the data sets we’re collecting. So, where’s the…what’s is the high value? Like, I would love to hear opinions about that.

Heckman: Well right now you’re not, I mean parents are part… you’re using them to leverage what you’re doing. I mean…well, I don’t want to focus…

Roberts: No, it’s ok. So, the short version is we are doing ethnographic studies of parents and their environment, but it’s not connected with this data and telemetry studies, so I won’t be able to line it up. We do have distributions and an idea of our audience, especially Title One and low income. We try to focus on the underserved, so we know a lot about what’s going on, but this particular data telemetry study isn’t lining up.

3. Designing Games that Encourage Pre-Readers To Create Online Accounts

Timestamp: 31:20

Clip here

Other Person: You might want to suggest…do students have free accounts where you can tell how they did on their first ever play of the game and then…Of course, you might probably have a selection of kids who continue to play based on them being good enough.

Roberts: Distinguished ability and identity is key to all of this work. But, you can’t ask a kid that can’t read yet or understand virtual identity to create a full account. At the same time, even kids who can, if you WANT them to create an account, it doesn’t mean that THEY want to create an account. So, they have to WANT to create an account. There has to be a motivational structure or incentive. And so there are a number of different strategies in play that all when taken together constitute our strategy, which is: what they’re able to do; what can we make them want to do; and now we have that level of distinguishability or identity.

So, we do have account systems. You can create full credentialed account systems-safe, private. Uh, but not that many kids do unless we go out of our way to make pieces of media that they can create themselves and then save. When they self-express their own identity through interactives and can save their own user-generated content then care enough to go do that kind of thing. But until we make something they understand and care about, it’s pretty much a non-starter.

4. Game Play Patterns and Social Emotional Behavior Tracking

Timestamp 44:40

Click here

Heckman: But what do you make of that? But just to relate to some of the other measures, if we go across the games these people are playing…is there some way of developing an inventory of social and emotional skills? Or…

Roberts: We are at the very beginning of this, the learner profile. I see it in two huge buckets. One is get all of the things you think might be relevant and co-locate them so that you can organize them by individual. The second is a free-for-all interpretation. I’m not sure what the strategy is there, but we will be able to compare, in theory, this game and that game, did they watch this video, did they NOT watch that video. What are all the slices and dices of what we think might be important and see what’s true about that subset?

Heckman: Is there any way to kind of project these people to get some baseline measure of some…We talk a lot about the Big Five or some other inventories, a suitable gauge. Is there some way to look at your measures and relate them for at least a subset?

Roberts: I think so. I think that if you have a player who consistently, across operations, different games, consistently exhibits powering through you might make an argument that that is something like conscientiousness. And then there’s all the grit, persistence, determination. Stuff like inhibitory control and self-regulation that are so important that I think plays into some of that. I mean the Big Five, each term, each label of the Big Five, subsumes so many constituent concepts. So, I think it would be about modeling the connection and being careful that you actually are only getting one time. But the end game here is that if you have enough tiny little pieces it makes for great evidence.

5. Tracking Behaviors in Games-The Tea Party

Timestamp 48:50

Clip here

Heckman: So far you’ve done it game by game. But I’m saying you link performance across multiple games for the same child.

Roberts. We’re working towards that. It’s specific to the game mechanics in each case; the interactions are situated.

Heckman: Couldn’t you design games that were sort of more intensive in conscientiousness? More intensive in curiosity, you know? I’m just using the Big Five.

Roberts: There’s a game we have that has nothing to do with math at all. You sit down at a big table, and all of your character friends are there for a tea party. You get to hand out cookies, and you get some feedback from time to time. You didn’t give any cookies to this person, and you know observe that… And we (the game platform) would know if you shared equally in the first place and if, when prompted, did you correct any inequity? And things like that.

Heckman: So does the tea party really capture interaction though among groups? I mean it’s a virtual tea party.

Roberts. Yes, these are NPCs, but do have a virtual world.

Heckman: And you still have a virtual player where you are interacting with a virtual bot.

Roberts: So we can model.

Heckman: Yeah

6. Branding to Get Kids to “Groove Out” on the Games

Timestamp: 1 hour 7 minutes 30 seconds

Clip here

Heckman: How do you entice the child to get into it? So…I’m just getting back to how easy it is to enroll…suppose you have a group of preschoolers that you want to target say in a Los Angeles community…how do I get all of them…

Roberts: It is intended primarily for the home use case, but if you were running a preschool and you wanted them to use it you would go to the app store…

Heckman: You can download them (the games), but just ignore them. So how do you get them to groove out on this?

Roberts: They have to WANT to. So, it’s a strategy involved in the design to make it appealing. Use characters that they love.

Heckman: And you’ve experimented with that. You know how to apply this stuff?

Roberts: The people who make the media, I think they do, yeah. They have a track record of great success in getting kids to care. We sink or swim based on the kids’ choices. If we don’t make something the kid wants to choose over Toco Loco (sp?) or other things we’re sunk. A lot of our effort goes into engagement strategies. In fact, our learning model has a giant engagement factor that multiplies everything. It doesn’t matter how good your instructional strategies is, if your engagement goes to zero, you’ve lost the entire game. Everything depends on getting the kid to care. We try to do it in trustworthy ways (laughter).

7. Digital Sticker Books, Behaviorist Incentives

Timestamp: 1:11:05

Clip here

Roberts: Am I able to take game play up to an assessment and predict the performance on the assessment? And can I revise that over time as they reengage? Because the kids will engage and re-engage and re-engage until they get bored or age out…So, the media strategy here is working in tandem with the instructional strategy.

Heckman: Did they have a set of goals within the, over multiple trials? Yes, ah. So they keep an inventory? So the incentive structure is cumulative…I’m now climbing Magma Peak, I guess a volcano, so I go up at 10,000 feet. You’re about to go to 11,000 or something…

Roberts: As you go, you’re unlocking pieces of a statue for a Statue Island that rises up out of the water. The statues are of your favorite characters and you’re unlocking sticker books and you’re unlocking additional interactives. And so as you go you’re getting immediate rewards and you can then build like a puzzle into your favorite character in addition you’re getting other things kids really love like sticker books. Sticker books are a BIG DEAL to a three year old.

Heckman: But do you give punishments, too. Suppose they fail? Does their island start to collapse? No, I’m just curious about your experimental incentive structure.

Roberts: I don’t think we punish them in the honest sense of that word, but we do give feedback intended to help them to go meta-cog about certain things that maybe they’re doing related to the game. So we can say, oh, I don’t think that’s going to work at all or that kind of thing.

8. Making Predictions From Tidbits

Timestamp 2:02:57

Clip here

Roberts: With the whole set up ultimately, what I’d like to do is have enough representations of force and motion; and enough representations of this math skill; and enough representations of the social emotional; all feeding into a vector of the learner that itself has a layer of interpretive layer on top of it that we could argue for years about all the hypotheses of how to take all the inputs turn them and do all the analysis for all of those high value inputs and determine what they are predictive of. So, I am not necessarily trying to maximize any individual outcomes at this time. I AM trying to make a system by which we have reasonable extraction of high-value tidbits about individuals, and we co-locate those things so that we can take that profile and try to do productive things with it.

9. Surveillance Via Facial Recognition and Age Detection Software

Timestamp: 2:03:50

Click here

Michelle LaMar: So that gets to something that I’ve been wanting to ask. How can you be certain what you are seeing is the child, a particular individual interacting with the app, rather than a parent helping them, a sibling, a friend…or is there some sort of co-op mode?

Roberts: There is definitely going to be some noise. We’ve done some prototypes on facial recognition and age detection. It’s early in that space. I don’t have a lot of confidence in it right now, but the idea was to determine whether there’s more than one peer or an adult and child scenario going on at the same time and passed back and forth. There are some self-selection strategies we tried that worked pretty well when it’s kids. They effectively go down the safe slots. The idea that you can create your own little avatar and you create your own slot and there are multiple slots available to you. Because when you give kids things they can create and save and care about, they want to keep them in their own toy box, that kind of thing. We’ve done some things to try and get the kids to tell us, but we may never solve that problem.

10. PBS Kids, A Huge Intervention

Timestamp 2:06:19

Click here

Brent Roberts: So let me just say I’ve loved you guys since I was a kid. You’re just awesome. You are, to be honest, a huge confound in every developmental study ever done. You’re like lead, right? Unmeasured, pervasive and having a huge effect on everybody; and we don’t measure.

So…if I am running a study, any study… at least half my kids are doing your product. You’re a huge intervention, right? Instead of thinking in terms of like how do you bridge from what you’re doing to doing clean experiments and or measuring things, because it seems really hard. You’ve got dozens if not hundreds of longitudinal teams in the field right now working with kids. If you can create a system where you simply give them in a research capacity the ability to tap into your system and track the kid, you get the IRB approval from the parent, the PBS folks will work in coordination.

So you don’t do anything; you don’t design studies. You just give your information to those researchers who can design studies, who are measuring these things. That’s the easier way to do this than to toil through all the different necessities. You got tons of people out there, and you are so pervasive. I mean I don’t see how a developmental researcher wouldn’t want to know.