導入とブラウン大学での出来事
Hi, Asami.
All right, so
What's new in today's
disappointing news
about the world at large?
I have mixed feelings on being assigned the roleof
disappointing world news deliverer.
It's okay. You also provided me with the LittleTall Man song to cancel out the effect.
Oh, yeah, that's right. I should I should I shouldalso probably yeah share that or like put it asput it put it in a
In a show notes. Yeah
Just for everyone's love it
It's going in the date column right now next tothe episodes list, but it's going it's going in myhead all day long
That's awesome
It's fantastic
Anyway before we before we do that thing where welike narrowcast because we haven't given anycontext for the audience
Um, yeah as to what that is. Don't worry folks.You'll have a link to it. You can check it out
Uh, it's the much better version of AI generatedmusic
and and that's that is
Sarcasm, I don't have a bell yet. We didn't buybells. So
Oh, yeah, I need to buy bells. I know
it's a it's a it's a
It's an artwork made by developing intelligence
Yeah, that's clever, which we very much
very much enjoy
Listeners are like what are you talking about?
What are we talking about what are we talkingabout I can tell you what we're talking about
So i'm going to give some context. This is goingto be about
a sort of recent i'll call it a follow-up toobservations that were made in
Uh april, uh, so roughly, you know springtime ofthis year where at brown university there were
Well, first off there was a terrible shooting anda tragedy which had occurred. Uh, so heads up,right?
um, we're not going to talk in detail about that,but that had been
Quite a disruptive and emotionally psychologicallyetc for students teachers faculty staff
and so
some faculty right had attempted to
Adapt to this and and well quote unquote say fromthis particular brown herald article right toalleviate pressure for students
Right end quote so something that would help themstill continue their learning in the face of avery tragic event
Fair enough, right because this was in a midtermseason. Yeah, exactly and this I find this sort of
Like the the timelines right and stuff. I'm onlypartially certain of but this was apparently forthe midterm
um
And it must have come fairly close after right thethe sort of december
yep, yep, so
um
I caveated that because since I spend all my timein the
Shifted academic university calendar. I sometimesget confused what counts as mid or end ofsemesters. So
I I had to sort of think about that for a second.But yeah, that sounds about right. So
What we have here is an attempt to give students atake-home exam
Instead of doing an in-class exam so that they areable to do it from a place of comfort at their ownpace
work through the material
Uh and and not have the added things like timepressure environmental pressure, right?
Uh, you know, it has to get done today. And so ifyou're sick and dying come into the classroom
Yeah, remove all of that and fair enough, yeah,which is very very reasonable giving that uh,
The sort of tragic event happened on campus andsome students might have difficulties coming tocampus, right?
uh to sit there in an enclosed room for hours andso, you know, I would say
This professor did a very reasonable and humanething for the students
to
Give them an option to take it at home and and hedoesn't typically do take home exam, I think
Uh for this course that seems to be the case. Sowe don't have to dance around it either
This is in you know, the articles that we'll list.This is economics professor, uh, Roberto Serrano
um, and they're, you know econ 1 1 70, uh, 11 70,whatever, however you pronounce that but
Uh, right
And they're not the only ones so in in this brownherald article that comes from the sort of aprilobservation
Here of of this attempt to give, you know, a take-home exam and alleviate those pressures
異常な成績と教授の対応
It had a really strange outcome
and
The outcome was that out of all of the years thathe had had these exams the average was maybe
Uh quote, you know 65 to 85
Right, that's a pretty tough exam. It's it's atough exam, right? It's you're not anticipatinggetting a high
score, right the exam itself is going to be
Probably something like a few professors. We knowof uh, we won't name names here
Oh, yeah, but like if you get a 70 you're veryhappy about yeah
You're very happy because like half of theproblems on there are unsolved problems in thefield, right? And like right, you know, so
Fine right i've never been one to consider thosethe you know, the most interesting of tests, but II can see it
I can respect the you know, right, right, right
Uh, yeah, and yet all of the students werebasically coming back with uh, 98 or 100
So, I mean the mode was 100 right? So the mode was100 the median was 98
So these which is why these were insane scores,right? This is basically it's just perfect acrossthe board
Right, right. Um
and this led to
clear skepticism about
How the homework take-home exam might have beendone
And of course ai is already in the sort ofeducational discourse and that was right the firstsuspicion
and they started observing things like
certain really strange or convoluted answers tosome problems and some of these
Answers were essentially identical to each otheracross students
and this is a sign that
Without an ai without a large language modelcatching myself there, uh without a gen ai or alarge language model
You know, maybe the students were copying off eachother's papers
But unless you are working with somebody who knowseverything about that course at the beginning
Then you're probably not going to be nailing, youknow a mode of 100 across your your class, right?
this is like your ta has to like
Distribute and make money
For the answer keys or something. Exactly. Yes.Yes, exactly
Um side note, I guess we we ought to have said howmany students this was out of so in the in thelater situation
Which we'll talk about they do mention how manystudents were there. It looks like roughly 59
Students except there are right 18 students thatdropped the class and nine that didn't take theexam
So it was actually bigger, uh, but then they rightso they started with 77 students
and
All 77 of them got really really high score in amidterm. That looks like suspiciously
Cheated. Yeah. All right, and then what?
um, and just as a sidebar in this, uh,
Uh brown herald daily herald article
Plenty of other teachers observing similarsituations, right? I think serrano's experiencewas
particularly
Poignant, right? It really stood out as like atake-home exam being so high versus other ones
but there were plenty of other professors andfaculty here who had a similar like
Our students just you know cheating on this, youknow by cheating. Are they just using genai?
Bypassing things are they breaking academic codes?
This is all that disruption and uncertainty we seewith
All of the sort of ai and especially ai and edtech
uh related discussions
and so
We come, you know months later and we head towardsthe the end of the semester and right professorserrano here
Uh says, you know what I as as we agree, right?
He did a very kind and caring, you know thing tohave them do this take-home exam, but clearly
Something was going on
and
decided that I am going to
Give my students a chance to quote, uh,
prove me wrong and quote that you know, thiswasn't just done by chat gpt or
Wasn't just bypassed in some way, right?
And in order to do that the final exam was goingto be in person
And he would compare the distribution of thescores from the final to the midterm
And you know assuming some some questions hereabout like, you know, how difficult were the examsthemselves but
Yeah, yeah
But I think it's a fairest thing he could do givenhis situation because he declared that to thestudents
Hey, i'm going to be comparing the scoredistribution between the midterm where you allsuspiciously did extremely well
And the in-person exam that we are going to takeas a final
final coming after the midterm
If you actually had a really good understanding ofmaterial up to midterm
You are at least going to score about 50 of theexam pretty well
You should be able to have continued the thegeneral understanding the foundation should befairly strong
Right. So like you're i'm expecting a similar kindof distribution if that is the case
And if that is not then we have things to talkabout right exactly. Yeah, um, and
uh, professor serrano, I think is is quoted hereactively basically saying
I expect that this is going to turn out to show methat you all cheated, right?
He doesn't say it that way, right? It's right,right, but like like no one does this kind ofballsy move without strong, right?
so
Um, and yeah, so they didn't get anything
Apparently the 18 who subsequently dropped theclass from this inside higher ed article that'stalking about, uh the final
Uh what happened and was published in july
That they dropped after
He stated this right that he would essentially oh,so they were like they they knew
They were like no. Nope. We're not prepared totake a final without that assistance
We know we bypassed it would be my guess rightbecause you hear your teacher go
Your midterm it used to count for this amount ofthe course
Um, right. I don't trust that anymore
And so you have to prove to me that you were infact learning if you were
If these are roughly similar, I will keep itright. I will keep it counting
But if not
I'm going to reweight everything right i'm goingto reweight it so that the final
The one where you're actively takes the mostamount of weight the most amount of weight exactlyand so 18 left
Uh, nine students also just didn't take the final.Um, so there's some, you know, probably caveat
So those people just admitted defeat basicallyunless you know, of course like they were likegravely sick or something. Yeah
Yeah, yeah, yeah, exactly. Yeah, so the resultsright so the results
AIの影響と大学教育の未来
Yeah
Staggering
Horrifyingly staggering now. There's there was animmediate thought that I had right when justlooking at these results
Which I think we've already covered right like ifyou give a take-home exam
You do expect or I would expect that the take-homeexams overall scores should be higher
Because you are given the time and the ability andhonestly
Probably still the chance to talk with yourclassmates a bit right and try to work some ofthese problems
Yes, when I had take-home organic, you knowchemistry exams for like the advanced versions, Imean it was yeah
They were like you can work with each other,right? It's hard. So I anticipate you strugglingand like then you will get exactly exactly
So I think I think I also had a quantum
two
Exam that was take home. Yeah, and
Like the teacher knew that it was difficult
so
Like he didn't have any problems with like usdiscussing
I mean there was no way he can like, you knowfollow around all of us to like make sure we'renot doing it
Yeah
but like we were you know, I think self-respectingenough that like we
might have discussed the problem, but we nevercopied each other off because
Yeah, you know that would be a surefire way to getboth of us in trouble
Right, you know aware. Um
We can anticipate consequences unlike
Children
Unlike children and I would say children is not anage designation. It's a general awareness mindset
Yeah, it's a mindset. It's a real big mindset. So
so basically what this distribution tells me isthat
Most people did wildly worse than when how theydid
In the midterm that was perfect score essentially
Yeah, and shout out to this one kid basically who
Did pretty similar to his midterm and the finalslike the one proof that you know, this person didnot cheat
Just just tried their best like, you know, thatwas that's what they were able to give. Um,
Yeah, and and there's also one one person whoseems to
Have really understood the same. Yeah
Really well in both situations
So like kudos, right exactly, but most everybodyhave more than 10 points if not 20 30 40 points
Difference. Yeah percentage points off from howthey did which was nearly a hundred right for mostof them
Uh in the midterm, yeah in a take-home, so that'sjust confirmed professor serrano's, uh suspicion
that uh
They did indeed cheat using some shape or form ofcheating method
and uh
And therefore, I guess their grades werereassigned based on
the new grading scheme
Exactly. Yeah
um
so so that was sort of like
The headline of the news. Yeah, the headline was a
You know a clear giant shift when they removed thechance where a possibly assistive tool could beincluded
which seems to be implying and
Is fairly likely that an ai tool was used toassist the take-home exams
I mean, we you know, we shifted from a 98 to a 48
Average school. Yeah, right. That's wild, youknow, uh, I think
Professor sonar even after that
The reweighting was the final becoming 80 percentof the students final grade
and so
Like it was a very it was a very I would say
stern, you know reassignment or reweighting of thethe score and
He did however make the passing, uh 40 percent.Um, like
So if they score 40 and they used to set it as a50 or higher, but I guess he's extending somegrace
Yeah, yep, so that was that was that's the resultright was
here we see the students and I wish
there was um
I don't think there is in this article. I wish
That there was some examples of like the thestudents even maybe maybe somebody who anonymouslyright was able to talk about
what had happened because
this experience must have been
Quite a rude awakening for the students as well asmuch as it was the the professor now
You know because it can't I don't imagine thatevery one of these, you know 60 to you know
80 students was like i'm gonna use chat gpt tocomplete my test
because
Because I just don't want to do the homework rightlike
right
That probably wasn't the motivation
You know, there are other things out there. And so
no, no, no and and so let's let's get talkingabout sort of like
What
Does this mean to the future of higher educationor you know going not even future just like acurrent situation
of
Uh higher ed like what does it mean?
What does it mean to go to college in the age ofai if your kid is just going to cheat his or herway out
for every single exam
Every single problem set and
Yeah, like what does it mean? Like does this ivyleague degree hold any value?
You know
This is this is a massive question which
you know three
Roughly three years ago right when this started tosort of pop up and it was like wow
Look at how fast and like smooth these these sortof text generation can be
That that everyone was starting to see and then Iwas
Getting upset enough to turn it into my wholefield of focus for you know a time being
yeah, but it was like
We have to ask this question
Because you we do you see it changing right? Imean like the rest of this article. It's alreadyhappening
It's already manifesting in this kind of behavior.Yes, and yes
And moss as well, like this is not just like 10people doing it. It's like the entire 60 70people. Yeah
Who are committing to this? Yeah
And and
you know, I don't know what their psychologicalstate was when they're doing this but like
You know, it's just like it's accessible. It'sdoable. Like what does that you know?
What should you do as a professor in
Another higher ed institution. Yeah. No seriously,right because this and
Even in the cases where you know, perhaps facultyand otherwise, uh were
Expecting it to to happen more slowly or
Just felt like they they weren't you know, giventhe time or the support or anything else
We see this here as well very likely that theywere not given any bit of support to reinventtheir syllabus or anything
No, um and technically universities are working onit, but they're they're behind I mean, they'rethey're not oh significantly lagging
yeah, yeah, I mean the
I
And my colleague in the ai and teaching practicecommittee that we were, you know running here forfor two years, right when I entered
We had some of the earliest like direct insightthat I felt was reasonable or or
Comprehensive enough to make statements on
And that was like, you know 2005 stuff, you know,and now we're starting to see
Uh, so brown university actually just releasedwhich I found as well
Their ai and teaching gen ai and teaching, uhcommittee report
For okay july and there is one person that we knowwho's on this committee. Um from the chemdepartment
um, and
uh, so
This was interesting to read but it's interestingbecause so they're they're doing a lot of mixingand like kind of trying to link
It to not just the u.s. But some of canada some ofthe uk
Uh, like, you know studies and surveys and they'vegot a lot of brown survey things in here
And it's as I said in our our very likely previousepisode the way that we're going to order thesethings
It's the same
It's this it's the same and I don't you know,people might nitpick me over this like oh, well,maybe it's like, you know
There's like 56 percent here and there's like 40percent, you know here in this community
That's not the point. The point is that the numberof people using the tools is increasing and ithappens to be increasing
Especially in educational spaces where we thenhave to address the presence that it exists
within that space or right you have the
not
Not absolute outcome, but you have the possibleoutcome of what professor serrano had to deal with
Um, right right like and you will get noassistance for it
So and i'm sure he doesn't want to do this, youknow, i'm sure he doesn't want to keep no, I betnot like
like
scaring the shit out of these students everysemester like
You know, i'm sure that's not what he like
Envisioned when he became a professor and startedputting together the syllabus
And and I think so two things right there's thingsthat faculty member
can and should probably do
and and also
things from the students perspective like
You know
It's it's a two-way street
I think you have a like you have already starteddescribing and also have a you know
very hands-on experience on what it's like to be
A faculty which again takes time to make a changebecause oftentimes
You guys are asked to make these changes with verylittle support
and
very
little resources as well
and
um
But it's it's critical. It's already too late andyou know
If we and and university is famously very slow onadopting any changes
and yeah
um
but like
it's
In this case sometimes for the good sometimes sortof bad, but in this case
It's kind of becoming a problem. Like, you know,what does it mean? Like are we gonna just admitthat?
the
GPAs from
2020 like what 2023 to like
2020a while we were still trying to figure out isvoid and completely meaningless because like, youknow
We were trying to figure out what this means,right? Like that's so unfair to everyone. Butyeah, so so
Things to do but takes time obviously. Um, so Iwant to think about more sort of like
students perspective right like if I
Were student now, which would be like we werestudents not too long ago, you know
Yeah, I feel like we narrowly escaped this uh,like
Crazy
Adoption of gen ai
Which i'm very thankful for and counting myblessings every day but um
like
I guess it's it's really
It's really strange it must be to be a student nowwhere
you
have
This person who has expertise in whatever he or heor she is teaching on
Uh have decades of training probably a phd also onwhatever they're teaching on
and
and also have this
really easy access to
Like what seems to give
Like same level of expertise
on a surface level
Right
and yeah
I think
I can see the temptation of like wanting to ask
you know internet for
um
the
Sort of you know the answers and you know notsaying that
When when we did problem sets when we were doingtake-home exams, we didn't rely on internet,right?
Like yeah, it probably like existed in front ofour eyes and we probably typed away and trying tofigure it out
what we didn't have was this like automaticallysynthesizing things that was like
Like shoving answers down our throat, right?
We still have to like vet our answers. We stillhad to like is this
like derivation actually correct because
There are wrong things on the internet
Yeah
Yeah, um do do sort of like
you know
and have a bit more sort of like we just like itdidn't occur to us that we would like ask chatgpdto like
You know copy and paste the questions and see whatlike spews out
um
And yeah, yeah, it just like
it
I understand the temptation but also
So
As what like
18 to 20 something year old
who has
Who are lucky enough to be getting universityeducation?
like why
Ruin your chance and actually learning
and
That is my biggest question i've got i've gotcomments, um, we've
It's it's a great question
Right, but that question
That's it for the show today folks. Thanks forlistening
Find us on max at egode science. That's e-i-g-o-d-e-s-c-i-e-n-c
See you next time