Hello and welcome to this edition of the What in the World is Dyscalculia podcast.
I'm your host, Dr. Honora Wall, and the podcast is put on by EduCalc Learning.
And you can visit educalclearning.com to learn more about the math learning disorder
Dyscalculia and see some of the products and services we have for teacher training and
for K-12 online classes for people who have Dyscalculia.
And we can also incorporate those into your school if you'd like to use them in the classroom.
If you visit the DTRI.org, you can get a lot of free information and resources from the
Dyscalculia Training and Research Institute.
And those are both places where you can ask questions about Dyscalculia, how do you tell
the math learning disability from other math struggles, and what you should do to help
support students, colleagues, and coworkers who have math learning disabilities.
Today I want to talk a little bit about testing, though.
And I want to talk about the new trend, the up and coming, all super exciting.
Everybody's doing it, even the SAT, which I found exceptionally bizarre, the idea of adaptive
testing.
I am 100% against adaptive testing.
I want to make that very clear.
So if you are a big fan of adaptive testing, this might be where you turn off this podcast
and go listen to something else.
I would completely understand because I can tell you I'm not going to be promoting anything
about adaptive testing in this episode.
So let's talk about what adaptive testing is.
This is a method of testing used predominantly with students.
Now we're seeing it sort of encroach into the higher education space as well.
It's very popular in math.
I don't know how popular it is in science or social studies or reading.
I haven't seen too many reports about that, but in math testing, it is the shiny new object.
So here's what happens in adaptive testing in a nutshell.
A computer is coded for test questions and the questions are coded for difficulty level.
And when the student takes the test, if they get an answer right, then their next question
is a little bit harder.
And if they get an answer wrong, then their next question is a little bit easier.
And the idea behind it is that the questions can go up and down a difficulty level finding
a maximum high point free student.
So this sounds wonderful.
We have a few problems though and a few questions that personally have not seen any answers to.
If you are in the world of adaptive testing and you have answers to some of the questions
I'm going to pose, please send the information to me.
I would love to know more about the coding process in adaptive testing because my first
question is what makes a topic easy or difficult?
How are we defining that?
Now of course the generic answer is oh we can define difficulty level based on grade
level standards.
Okay, super.
How do we know whether or not a question is designed to interpret or analyze proficiency
on a standard?
Now this is a huge question.
I do not have time in a podcast series much less one episode to answer just that question.
The idea of validity and reliability, these are very specific terms when it comes to assessment
tools and when it comes to research.
And it is not something that can be done off of one question.
One question bank, one test writer, the whole purpose of validity and reliability is that
something tests exactly what it is supposed to, not anything else, and that it gives equal
results across a large group of people.
So that brings me to another question.
How can we say that any of these adaptive testing tools are yet valid or reliable when
they are so brand new?
We need so much data before we can rely on the results of adaptive testing that at this
point in time it is really a data gathering tool, it is not an assessment tool.
That is my first two questions.
My next question is how do we know whether or not a question is measuring what we want
it to do?
And this is actually a very involved question.
There will be some longer episodes because this has happened a lot in my work with students
and I need to address it in some YouTube videos so I can show you the word problems and the
formatting of questions and answers so that we can have a greater discussion.
But there are a lot of assessment questions out there that are so convoluted, so dense,
they are really testing vocabulary level, they are testing sentence complexity more
than they are testing math knowledge, computational or conceptual.
This is a problem not only for people with dyscalculia but also people with dyslexia,
people with ADHD, people with dysgraphia, people with different educational backgrounds
based on how much vocabulary exposure they may or may not have had at their school and
that happens in all levels of schools not only low SES but also any school of any size
because it is really teacher based as well as curriculum based, it is very complex.
So we need some assessments that will reach a much larger audience not only the kids who
have had tons of exposure to higher level vocabulary and who read incredibly well and
who interpret their text very well.
We also have a lot of questions that are working backwards if you know the total volume how
do you work backwards to something, if you know the total of this how do you work backwards
and then use it to find the whatever of something new with a change.
And those are great questions and absolutely students with dyscalculia are more than capable
of answering them if they have had the exposure and the training and the practice and the
work to prep for that.
So we really don't know that going into assessments and a lot of math assessments if you look at
them they don't have a really wide variety of levels.
And what I mean by that is let's say we do have a question where we want the student
to take the volume and work backwards to find let's say the side or the area of something
or the height or the radius of something.
Okay so we need them to use some inverse operations and work properly within a formula.
So then a test, adaptive or not should really test that concept in a variety of ways with
a word problem with higher level vocabulary, with a word problem with more accessible vocabulary,
with an image, without an image, with a formula listed, without a formula listed, with a calculator,
with numbers that make sense, with a fraction, with a decimal, with whole numbers.
That way you are giving the same topic, the same skill set of finding one particular measurement
when given the total volume.
And that's what you are searching for.
With a variety of question types then you can decide, alright this student performed
very well with an image.
This student performed very well when there was no clutter and they just had a very direct
word problem to answer.
This student performed very well when they were given information and had to find the
total volume.
This student worked very well when they had volume and they were finding a side.
But maybe they didn't do it as well with radius.
Okay now we understand there is a vocabulary difference or some intimidation with circles.
So having one skill set presented through a number of different question types, now we
can really find out where the problem may or may not lie for individual students.
I have not seen any evidence of adaptive testing approaching assessment in this way.
In one skill set, one topic and presenting it in different ways.
To find out does the student have an issue with word problems, with graphics, with fractions
or is it a problem with understanding how to work the formula.
And that is a really important question.
And when we are talking about getting a more difficult or less difficult question, because
a student got the answer wrong.
Here is where I really have a problem with adaptive testing.
Why did the student get the answer wrong?
Now everyone listening probably has their own idea in their head about why a student
got an answer wrong.
If we could put you all in a room, you would all have a different comment on why a student
got an answer wrong.
And that is exactly the problem.
We don't know if the student has never worked with volume, maybe they don't know their basic
facts, maybe they freaked out at the word problem, maybe they have terrible anxiety over test
taking, maybe they have a cold today, maybe their parents are getting divorced or they
just moved or in a lot of places across the country.
And any time during the school year there could have been a hurricane, a tornado, a
thunderstorm with hurricane force winds that destroyed something in town.
There could be a family medical emergency.
All of these things impact performance.
Maybe the student got the answer wrong because they hit the 7, but their finger also hit
the 4 and they typed in 47 or 74 without realizing it.
A lot of these adaptive tests are moving away from multiple choice answers.
They are only giving a blank fill in answer.
Well then, we have no idea if the student rounded incorrectly, estimated improperly,
was the computer program coded to take a variety of answers in case the problem was estimation.
Maybe the problem is that the student forgot to hit a decimal point or put it in the wrong
place, which is an incredibly different problem than not understanding the math topic at all.
So we can give these adaptive tests, but all we know is that we are bouncing children around
like ping pong balls.
We have no idea why they actually performed the way they did.
We don't know where their skills are, where their strengths lie.
We don't know what the weaknesses or what the problem is if they don't perform well.
We don't know if they're just a really excellent guesser.
And we don't know if they were just hitting in random answers and happened to do it well
enough to score at a certain level.
We really have very little understanding of what makes an answer right or wrong.
And a lot of that can come from the student, a lot of that can come from the environment,
and a lot of that can come from how the computer program was coded.
When you create an assessment on a computer, you have to put in the problem and you have
to put in the acceptable correct answer.
There's a lot of times in math when we round to a near whole number, we round to the nearest
tenth or hundredth, and we don't know if the computer was programmed to only round to the
whole number or if it was coded to take answers that were rounded to the tenth or hundredth.
We don't know if there's any leeway in acceptable answers.
And that's a problem.
If something is valid and reliable, it doesn't need to be hidden.
You can tell people exactly what's going on.
You can code four answers that are one digit above or below for an estimating problem,
and you can even note that.
Do adaptive tests do that?
I have no idea because it's very difficult to find out this kind of information, at least
from the places I've looked.
As I said, if you happen to be in the world of adaptive testing and you have some more
information, please send in my way and I will be happy to share something from the horse's
mouth, get it directly from the source, and put that information out there.
But when we don't have information, that does not build trust.
That does not build validity or reliability if you're hiding something.
And that is a real big problem that's been prevalent in math assessments, especially
for quite a number of years, especially with state testing, those end-of-course exams,
state-based tests at the end of the year, and now with adaptive testing, I'm very concerned
about how this might play out in the world of college level placement tests.
And that problem is that teachers are not supposed to glance at an EOC or a state assessment.
Students take that.
Teachers are not supposed to know what's on there, not supposed to look at it at all.
That's ridiculous.
First of all, I find it insulting for educators.
We are trained professionals.
This is a career choice.
This is not something we do on Tuesdays.
This is something that we have higher level degrees in and a number of years of training
and we know what we're doing.
And also in the world of math, it's just math.
There should not be any reason to hide an assessment from a teacher.
Teachers should know exactly what their students are going to be expected to do and how they
will be asked to do it because there are very complex differences in a word problem, in
presentation, in understanding what is expected and what's the best way to solve a problem.
Adaptive testing, especially when you can only see one question at a time, especially
when there are no multiple-choice answers, multiple-choice answers do make things easier
than a fill-in test.
And I'm quite sure that's why adaptive tests and other assessments are written that way.
However, multiple-choice answers also trigger memory.
They help people think about their potential answers, the potential problem-solving skills
they might need, and then they can determine what is best for a given situation.
So not only are we taking away from student guessing, I get that we don't want students
to guess.
I think it's a little ridiculous because we all guess and estimate all the time.
The human brain likes to ask questions, predict, and then confirm.
This is how our minds work.
So there's no reason why our assessments shouldn't work in the same way.
Also, putting students in a situation where they have a few answer-choice options can
help them think about, okay, how could they get this answer?
This one they were adding.
This one they didn't combine like terms properly.
This one is close, but the decimal points are all in different places.
I need to focus on the decimal point.
So these are guidelines.
The same as using a map, the same as having a sign on a highway that tells you what exits
are coming up next.
It gives you a frame of reference.
And I don't see why that's something we should remove from K-12 students.
I don't see how that makes your understanding of math any stronger or any weaker if you
prefer to have some guidelines to make sure you're going down the right path.
So that's another problem I have with adaptive testing and only fill in the blank testings
that are removing multiple-choice answers.
So in general, I have a lot of concerns about what's happening in the world of assessments.
And this is a problem because so many colleges and universities use SAT test scores for acceptance.
They use these placement tests to decide what kind of course a student should take.
We're using a lot more online only math courses in higher ed.
And we just don't know what's really going on with our students when we do this.
It's almost like a weird personification, putting these human abilities onto a computer.
That is just not where we're at, folks.
Even if the AI tools made the most amazing adaptive testing situation in the world, until
you talk to a student, until you get to know a person as a human being, you're not going
to know exactly what happened to make them get a right or a wrong answer.
I'm a big fan of technology.
I do have online math courses for students.
It's not like I'm a Luddite, but I am saying let's make sure that our assessments truly
align with the instruction, which is kind of the point of an assessment.
It's not to find out whether or not you know how to cook an egg.
It's designed to find out whether or not the course you took taught you how to properly
make a souffle versus how to properly make a fried egg sandwich.
And can you interpret when you should be doing which one based on what someone orders
at a restaurant to use a restaurant analogy?
These are complex questions of assessments, but they're mostly complex because we take
so much of the humanity out of math assessments.
That's really a problem.
At the end of the day, we want students to feel comfortable with numeracy, comfortable
with numbers, and comfortable working with formulas.
We really want to strengthen pattern recognition and problem solving.
So let's focus on making those things happen, and I don't see how adaptive testing is anywhere
in the same ballpark as any of those goals that I just mentioned.
Now I'm sure other people have their own opinion, and like I said, if you have some research
or some facts and figures to back it up, then I would love to have that conversation.
If you do, you can email me, Honora, at educalclearning.com or Honora at thedtri.org.
If you are interested in this conversation, let's talk about it some more.
You can talk about it with me, with your colleagues, with teachers at school.
I would love if you had this conversation with your local school board members and with
your state representatives and your State Department of Education.
These are really important questions for us to answer.
We do have problems with math awareness and math proficiency, and honestly we can fix
them pretty easily because it's just math.
And all humans are hardwired to understand math, even those who have dyscalculia.
All people are hardwired to be able to see, and some of us need to wear eyeglasses, and
some of us have more extreme vision problems, and we can still use the right tools to have
a full, rich, involved life.
That is the dyscalculia piece in this whole conversation.
We need to understand that some people have a different approach to math understanding
and math proficiency.
Different doesn't always mean worse.
It doesn't our present school system, but it doesn't have to.
Anyway, I got a little off topic there at the end, but adaptive testing personally for
me that is a no-go, and I really would like us to have more conversations about creating
appropriate math assessments for the people taking the math assessment.
Not something that looks really good to the people in the company creating the math program.
This is Dr. Honora Wall, and I thank you for listening to the What in the World is Dyscalculia
podcast, and I hope that you get more involved in a rich conversation about math instruction,
math proficiency, and math assessment.
It's important.
Thanks for listening.
I will talk with you again soon.