Psychological Testing in a Custody Battle with John Zervopoulos | Episode 118

The subject of parental responsibilities evaluators (PREs) and child and family investigators (CFIs) have been touched on in our podcast before. Today we welcome back John Zervopoulos to do a deep dive on these areas of psychological testing and important things lawyers need to know about them. John does a great job of contextualizing the comparing of CFIs and PREs as well as laying out the different responsibilities of lawyers and psychologists in this regard.

If you have not yet listened to our previous episodes with John, ( Episode 23 - Understanding CFI or PRE Reports in a Colorado Custody Battle, and Episode 50 - Litigation Tactics for CFIs & PREs) we provide more of John's background and some important introductory content to today's conversation. In today's show, we discuss six keys that lawyers can use to manage an evaluator's test results and we go through this list, with John weighing in with many expert tips and reflections, before ending off our chat with some thoughts on other kinds of tests, the influence of context, and more. 

Key Points From This Episode: 

• Unpacking the two different avenues of psychological testing; CFIs and PREs.   
• The actual nature of the tests; comparison and incorporation with other information.  
• How psychological testing differs from other screening processes. 
• The specific elements and methods of PREs; John unpacks his 'three-legged' approach.
• How the pandemic and the remote landscape have affected how tests have been administered.  
• Introducing the six keys for lawyers to look at when considering the evaluator's use of the test results. 
• The importance of the response style of the examinee.
• Significance and meaning around standardization samples of the test. 
• Laying a strong foundation with the test's statistical reliability.
• Ways for lawyers to gauge a test's validity.
• How sensitivity and specificity can aid our understanding of the accuracy of the scores.
• The three-step process of computer-generated interpretive reports. 
• How to reconcile the possible use of different tests by psychologists.

What is Divorce at Altitude? 

What is Divorce at Altitude?

Ryan Kalamaya and Amy Goscha provide tips and recommendations on issues related to divorce, separation, and co-parenting in Colorado. Ryan and Amy are the founding partners of an innovative and ambitious law firm, Kalamaya | Goscha, that pushes the boundaries to discover new frontiers in family law, personal injuries, and criminal defense in Colorado.

To subscribe to Divorce at Altitude, click here and select your favorite podcast player. To subscribe to Kalamaya | Goscha's YouTube channel where many of the episodes will be posted as videos, click here. If you have additional questions or would like to speak to one of our attorneys, give us a call at 970-429-5784 or email us at info@kalamaya.law.

************************************************************************

DISCLAIMER: THE COMMENTARY AND OPINIONS ON THIS PODCAST IS FOR ENTERTAINMENT AND INFORMATIONAL PURPOSES AND NOT FOR THE PURPOSE OF PROVIDING LEGAL ADVICE. CONTACT AN ATTORNEY IN YOUR STATE OR AREA TO OBTAIN LEGAL ADVICE ON ANY OF THESE ISSUES.

Transcript

Ryan Kalamaya (3s):
Hey everyone. I'm Ryan Kalamaya

Amy Goscha (6s):
And I'm Amy Goscha

Ryan Kalamaya (8s):
Welcome to the divorce at altitude. A podcast on Colorado family law

Amy Goscha (13s):
Divorce is not easy. It really sucks. Trust me. I know, besides being an experienced divorce attorney, I'm also a divorced client,

Ryan Kalamaya (20s):
Whether you are someone considering divorce or a fellow family law attorney, listen in for weekly tips and insight into topics related to divorce co-parenting and separation in Colorado. Welcome back to another episode of divorce altitude. This is Ryan KK in previous episodes, we have talked about Pires and CFIs, and that stands for parental responsibilities, evaluators, and child and family investigators, and really they are expert witnesses in parenting disputes. And we've had guests talk about what goes into those sorts of evaluations and our guest today.

Ryan Kalamaya (1m 4s):
John Zervopoulos has actually been on the show a couple times before and episode 23, understanding CFI or P E reports in Colorado. Custody battles is our most popular episode ever in the history of this podcast. He is also along with me in episode 50, where we break down for the family law Institute, which is the annual divorce lawyer's conference about parenting responsibilities, evaluations, and, and CFI reports. And so we are really lucky and, and we think that you are listeners are going to enjoy today's topic where John really dives in and dissects psychological testing.

Ryan Kalamaya (1m 52s):
And if you haven't listened to those previous episodes, I would highly encourage you because we introduce John and give a little bit more background. We're gonna skip over some of that today. It's gonna be referenced in some of the show notes, but also just there's a understanding of what we're actually talking about. So this is kind of some advanced content and topic above and beyond what we've covered in those topics. So without me dragging on even further, John, welcome back to the show. Thank

John Zervopoulos (2m 23s):
You so much, Ryan. I'm so pleased to be here. So before we get going, let me just kind of give you a little bit of a background to set the table for what we're gonna be talking about. Okay. I'm both a licensed psychologist and a licensed lawyer. So I bring both disciplines, the issues in both disciplines to the table. I did PS or child custody evaluations for several years and then went into stop, shifted my practice to doing consulting with lawyers. And basically what I do is I help lawyers understand mental health issues in their cases and assist lawyers to manage the work and testimony of mental health experts.

John Zervopoulos (3m 5s):
And so it's from that background that I am able to talk about the testing various aspects of PRS from both the psych psychologist aspect, but also from the, from what a lawyer would need to know and how that can affect and frame the way they deal with their case.

Ryan Kalamaya (3m 29s):
So, John, we're gonna talk about psychological testing and again, to kind of further set that table listeners should understand that in Colorado, there are two different, you know, main types of evaluators. There is a child and family investigator, and then there's a P E and Pires are allowed to do psychological testing by law. CFIs are, are not allowed to do testing. And so, but they will sometimes outsource that. They'll bring in some mental health professional that is licensed and allowed. And there's a lot of kind of mystique and misconceptions about psychological testing. So John, for listeners that don't know, can you tell us, what do you make of psychological testing and what are the kind of two different angles that really exist out there on the viewpoint of the psychological testing and, and how useful it is?

John Zervopoulos (4m 22s):
Well, first of all, it's important to realize that when a person is appointed to do a P E in Colorado, right, they don't have to be a psychologist, you know, licensed professional counselors, social workers, non psychologists can also be appointed to do PRS and only psychologists do psychological testing. So the question is really, you know, what to make of psychological testing because different disciplines are enabled if they're qualified by the statute, by experience so forth, to do PRS, not, you know, testing is often sort of dismissed as something sort of like a niche sort of deal.

John Zervopoulos (5m 2s):
And because there's some sort of mysterious stuff to it, people, you know, people have different views as to how helpful a testing could be. Of course, non psychologists, they weren't trained in testing. That's the part of the training for a psychologist. So non psychologists say, well, we don't really need to do testing. And they sort of raise kind of a red herring that there's no test that has been validated to predict post divorce, parenting arrangements. For example, there's no test that says that 50 50 is the best arrangement or alternate weekends is the best arrangement. So they say, well, why use tests? And, you know, my response is, well, tests can be effective and useful information for a P E but not for the purpose of deciding whether a particular schedule is useful, but rather to give information about the, the parents and sometimes the children that's incorporated into their conclusions.

John Zervopoulos (6m 2s):
Lawyers also have this sort of mystique about testing that, you know, you're dealing with interviews, everyone's asking about opinions from the experts and so forth. And now suddenly we have something with numbers. This is scientific and lawyers will kind of jump into that. Okay. But the problem is is that if the testing comes out, seems to come out against your client, then the lawyers sometimes dive into questioning the expert about the testing. And that is almost 100% a losing proposition. If you don't have an expert working with you, why? Because in law school, we never got a course on testing, all right?

John Zervopoulos (6m 45s):
But psychological testing is baked into the training for psychologists. And even if they did a poor job, they know the jargon, they know the language that will be able to pair with you on any examination questions you think might be a great question, but everything's gonna be muddled and you're gonna lose a lot of time. And possibly the, the Goodwill of the judge who also will be sort of confused about what all this testing means. So our thought was today, as we were talking, Ryan was okay, we don't wanna really get into the weeds in terms of testing, because again, lawyers didn't take testing courses, but there's certain keys that lawyers might ask about in a deposition, probably not in cross examination, unless you already know what the expert's gonna say, but at least some of these questions can give you an idea about how the expert is using testing and whether they're using it appropriately for, to inform their conclusions and decisions.

Ryan Kalamaya (7m 52s):
Right. And I think it's helpful even for listeners that aren't lawyers, but if they're going through an examination, a P E or a CFI, or they're contemplating it, they hear about the psychological testing. And I, I can speak from experience. John, when I was a younger lawyer, I, I had clients that said, you know, the other Eric Wolf, he's, he's bipolar, he's a narcissist. And I would respond well, they have these, you know, these, the, these exams and they'll do the multiple choice and they'll fare that out. It's like an x-ray. And so can you tell listeners why I was, you know, wrong and, and then we'll move on to the three testing issues in terms of the basics to give people a feel for what all goes into testing and how they can be used in an evaluation.

John Zervopoulos (8m 44s):
Yeah. Well, first of all, it's important to realize that a test psychological test is not an x-ray. All right, you break your arm, you go to the hospital and the x-ray shows the broken arm that's real clear. Okay. There's not, not very much dispute about that. All right. But a psychological test is very different. A psychological test, as we'll talk about later is comparing the examinees results to the results of other folks who took the test before. And, and to see if that examinee is, can compare to those person. And from there we say, that person is like the standardized sample.

John Zervopoulos (9m 26s):
Okay. And we'll talk about that in a few minutes. But the bottom line is, is that while with a physical x-ray, you can see the broken bone and right away make the correct snack job. This is a broken bone, psychological testing information has to be incorporated into other information that the evaluator gets through interviews through of the parties and, and, and their interviews with the children, through the collateral materials, you know, the materials that the evaluators looked at, the case facts and so forth, and the psychological testing, the three frameworks for an evaluation testing, doesn't stand on its own like a physical x-ray might.

John Zervopoulos (10m 10s):
It really is something more. And that's one of the things that we wanna take away from what we're talking about today.

Ryan Kalamaya (10m 18s):
Yeah. So, John, what kind of tests are we talking about when we say psychological tests? You know, like it's not an x-ray, so, but what exactly are we talking about here? What, what are the options?

John Zervopoulos (10m 29s):
Just because a psychologist gives a questionnaire, asking you to answer true or false, or ask to answer on a rating of one to five, or what have you doesn't mean? It's the, a psychological test, right? A reader's digest type up test to see if someone is, is depressed. That's not a test. Either a test is a set of questions. And again, we'll go into that when we get into the keys that has been set up, that is more than just, let's say in depression, do you get sad a lot? Are you a blue a lot? Do you try? Do you have negative thinking and so forth? Rather the testing, those questions are developed in a sci more of a scientific way that compares with people who have already been identified as having these traits.

John Zervopoulos (11m 18s):
And then if you answer the question similar to them, then you are deemed as being similar to them. So there's an inference there, and that's kind of how the UN the interpretation of the testing results go by. So it's not a reader's digest. It's also not a screening sort of test, for example, there oftentimes in Pires, there's a question about substance abuse. All right. And what the evaluator may do is do a, an interview of the parent and then give them what's called the sassy, the substance abuse, subtle screening inventory.

John Zervopoulos (11m 58s):
And then the person answers the questions on that screening inventory. It takes about 10 to 15 minutes to answer it, and then puts the results together with the interview of the parent. All right. And of course the parents not gonna say a whole lot bad about their substance abuse in a P E right. And then the evaluator makes a judgment because of the sassy instrument that the person is, does not, or does have a substance abuse problem. The issue though, is that in the very title, although it looks like a test it's called a screening inventory and in the instructions for interpreting the inventory, it says that it, the inventory should not be used to diagnose the person with substance abuse problem, but should be used in, be informed by other information that the evaluator gets in order to finally make a decision about SEP abuse.

John Zervopoulos (12m 57s):
So that's not the kind, that's not really a test. It's a screening inventory. The kinds of tests we're talking about are the P two, the MPI two RF. Now the MPI three is out a test like the personality assessment inventory, the MCMI that's the mul, the Milan clinical multiaxial inventory, three or four certain neuropsychological tests, and some intelligence tests, too. Formal intelligence tests are the kinds of tests that have been formally developed and standardized to be. And there are others as well, but to be classified as a psychological

Ryan Kalamaya (13m 34s):
Test and John, as far as I understand it, correct me if I'm wrong, the MPI, and some of these other tests, it's like multiple choice, get into kind of what all goes into it. But is, is it fair to say that there are multiple choice, many of them

John Zervopoulos (13m 49s):
Test? Yeah. MPI is actually a true false test and like the, as, as well as the MCMI that PAI, I think is on a 1, 2, 3 scale. So pretty definitive in terms of the answers to the particular questions

Ryan Kalamaya (14m 5s):
And John, how do psychological tests, how these fit in a P conducted by a psychologists who's qualified to administrator the NPI or the personality assessment?

John Zervopoulos (14m 17s):
Well, as I mentioned before, there are different elements of a P E. And we talk about that in one of the other podcasts in, as we talk about Pires, but to quickly go over that, the way I look at methods for P E is to view the methods as a three-legged foot stool. And one leg is the interviews of the parents and the parents and the children. The second leg is the psychological testing in questionnaires. And the third leg is the collateral material folks that the evaluator talks to. Let's say teachers, counselors, and also documents, court related documents and other documents that the evaluator reviews, all right.

John Zervopoulos (15m 0s):
And all of that goes into a good P E and metaphorically speaking. And this is a good in terms of preparing for arguments in to the court and so forth. If one of those legs is weak, okay. Then the footstool begins to wobble. If two of the three legs are weak, then that footstool has real problem staying stable, all. So being able to picture the evaluation in that way gives you a sense of how strong that evaluation is. So psychological testing, being that one leg it's informed by information from the other two categories, and it contributes to the final opinion in that way.

John Zervopoulos (15m 43s):
And that's the, that's the kind of, that's the fit that it has in a PR E.

Ryan Kalamaya (15m 48s):
Okay. So let's drill down on that and let's peel back or pull back the curtain on the wizard of Oz. They kind of apply a metaphor here, John, what do psychologists do when they actually conduct the testing? So let's pull back the curtain from the wizard of Oz, and let's see what's going on behind the curtains here. What do they actually do?

John Zervopoulos (16m 11s):
Well, first they administer the psychological tests. They have the person sit in a room in their office or be observed by someone in their office and the person sits and takes the MMPI or the PAI or the CMI. And then the psychologist then interprets the test results. And again, it's beyond merely getting a computer readout of possible tests, result. Interpretations. We'll talk about that. One of the keys where the MMPI, the CMMI and the PIF for that matter offer the psychologist, a computerized readout of the results of possible, you know, indications of possible concerns that the test shows okay.

John Zervopoulos (16m 55s):
That, that are developed through the computer and the like test results do not, as we said, do not provide an x-ray. So what like a physical x-ray. So what psychologists then have to do is integrate the material with case facts in issues. I think there's one quote that is particularly relevant and it's from the standards for educational and psychological testing. And this is a document that's put out by the American psychological association and two other entities. And the quote goes like this. When making inferences about a test takers past present, and future behaviors and other characteristics from a test score, the professionals should consider other available data that support or challenge the inferences.

John Zervopoulos (17m 42s):
Okay. So there's a couple things there, again, as we've talked about before other material information's important to interpret psychological testing. The other issue is what's going on in the psychologist's mind, we're talking about professional judgment there. The psychologist is, has to be in the process of themselves, interpreting all the information and incorporating the psychological test to get their opinions. And so that's kind of what goes on when the psychologist conducts the testing and testing process.

Ryan Kalamaya (18m 14s):
Now we talk, we're gonna talk about kind of some six keys. And, but before we get there, John, you mentioned, you know, that psychologist, the P E he or she will administer the psychological test and you referenced it being in an office. So what actually happens when they administer the test, do they, can they give it to 'em? Can it be like a take home test?

John Zervopoulos (18m 34s):
Well, generally not. Okay. Because for a couple reasons, the tests are not really standardized to be take-home tests. Although I think there's some research that's talked about the MPI being, allowing people to take the P home and so forth. Those are very different conditions, right? Those are conditions where the person is in therapy and has an incentive, to be honest, when they are answering test questions, someone in a P E situation, they're gonna go home to their kitchen's table, you know, maybe, or a glass of wine, have their significant other next to them and be answering questions. You know, I wonder this, that, and so on.

John Zervopoulos (19m 15s):
That's not standardized at all. So it's gotta be in the office. Right. And it has to be in the sense, in the same manner and under similar conditions by which the test was administered to groups of people when the test was developed. Okay. So that has had some interesting layout in, you know, within the last two years. All right. For example, even though tests were given in offices, the last two years, evaluations were done over the internet, all right. People didn't take tests in psychologist office because psychologists didn't go to their offices. All right.

John Zervopoulos (19m 56s):
So having, making sure that the tests were administered properly is very key. And now as we move out of COVID all right, and people have, and there's been some studies that have done about doing testing away from the office. I think it's important for lawyers to realize that where the test is administered is pretty key. And that should be top of mind in terms of some of the questions. All right, second then after admission test, the examinee responds in writing or verbally to the test questions that are listed in the test booklet or verbally offered by the examiner.

John Zervopoulos (20m 37s):
Okay. That seems pretty straightforward. But what if, and this happens, one of the test take one of the parents has had a Le a reading disability, and that's been, that was a problem when they were in school, it's still a problem. As they're an adult, the MMPI is a pretty long test, 567 questions. And for someone with a reading disability that can take a long time to get through, not to say how they might be misread or misinterpreting certain questions. Right. How does the evaluator handle that? So that'd be another question right now. It just so happens that the MMPI, for example, supplies a tape with someone maybe like a Siri voice or an Alexa voice, or reading the questions, but let's say the evaluator doesn't have that tape and instead has himself or herself or their assistant read the test questions.

John Zervopoulos (21m 34s):
Okay. Is that proper? Can that affect the test scores? All right. And then the examine, these test responses are then compared to answers from the identified groups of people who are part of the group that developed particular scales on the test. And we'll get into that when we get into the keys.

Ryan Kalamaya (21m 55s):
Yeah. So let's get a little deeper here and talk about the keys that evaluators must consider when they're interpreting the test results. Cuz we first talked about how they administer. Now let's talk about how they interpret. And I think it's helpful for, you know, lawyers and people going through. They can just understand the process of a, of a P E so that when they're going through the, you know, the evaluation or it comes out and there's a report that people know what to look for. And, and the first one John is response style of the examinee. So how is the examinee responding?

Ryan Kalamaya (22m 37s):
And, and tell me, talk to me more about what the key is in for key number one.

John Zervopoulos (22m 42s):
Well, be before I get into that, let me almost, in a sense, if I could Ryan switch the emphasis, you know, you mentioned about what the evaluator should be looking for, what, what I would like as we're going through, this is what the lawyer should be looking for. Because as I said, getting to the weeds with evaluators and arguing with evaluators is very problematic, but these keys are six things that a lawyer might look at that can right away give some sense of whether the evaluator is dealing with the test results appropriately. So the response style of the examinee is key one.

John Zervopoulos (23m 22s):
And the issue here is what is the examinee's approach to the testing? All right, now the MPI two and some other test scales have other tests have scales called validity scales. And these scales purport to reflect whether a person is trying to look good on a test or trying to exaggerate problems. All right, right now, clearly in child custody evaluations, a person's trying to look good, right? They don't want TOA. They don't want to mention problems that they're having because you know, an evaluator's their sense is that an evaluator's opinions at the end are gonna be very persuasive to the court.

John Zervopoulos (24m 4s):
So they're gonna try to look good to contrast that, imagine like a criminal case where you have an insanity defense and the defendant is taking him MPI. And the question is, you know, what was his mental state at the time of the offense, the alleged defense, right? Well, he may be trying to look bad, right? To show that, you know, he really has had problems. So this is the issue here. What is the approach that the evaluators that the examinee is, is using keep in mind too, that real quickly, that it's the validity scales, which are not really constructed real well anyway, they are harder to interpret when a person's trying to look good than when a person's trying to look bad.

John Zervopoulos (24m 55s):
Why? Because, you know, we're socialized to present our best selves to people from the time we're little kids, our parents tell us, you know, smile, be kind this and that. We know how to look good, but few people know how to look bad in terms of pretending that they're hearing voices and that sort of thing. We don't have very much experience with it, child custody situations. It's harder for the evaluative tease out whether the person is, is really trying to exa to, to look extra good. Or maybe because they're a bit shy in terms of not wanting to answer personal questions.

John Zervopoulos (25m 37s):
So the key que the key takeaway with that is what are the examinees most motivations for answering the test questions and keep in mind also that just because the examinee may not may have a high rating on that particular scale does not mean that they're not truthful in other parts of the evaluation too. Oftentimes evaluators take that reading on the MMPI or the MCMI and say, well, the person's just not being truthful, or maybe they're just cautious about talking about personal problems in the MPI, but they're truthful in other areas and interviews and so forth. So be real careful in terms of the response style.

John Zervopoulos (26m 19s):
So that's key one.

Ryan Kalamaya (26m 21s):
So key two John standardization, sample of the test. Tell us about key number two. Yeah.

John Zervopoulos (26m 28s):
Key number two is significant because it really kind of tells you the meaning of the scores. All right. Ultimately what you have is a standardization group around which the test was developed. Let's say that a person wants to develop a, a test on depression while they develop. They develop a bunch of questions. All right. And they're not just gonna say, well, this is a test of depression and give it to the person who's taking the test. They have to find people who are already diagnosed as depressed. Give those folks that test, see how they do, and now take the test to the person who's taken the PRA P I.

John Zervopoulos (27m 10s):
And if that person scores similarly to those persons in the standardization sample, okay, then, then we probably have depression, although I'm simplifying it for the sake of getting the idea across. But the notion is that there has to be a group from which that concept comes from for the test taker to be identified with that particular group. So a key takeaway on that is that mental health experts too often gloss over standardization considerations. So an important thing for lawyers to do is to check and see if there's anything particular to the test standardization sample.

John Zervopoulos (27m 55s):
Here's an example, consider an instrument that suggests that a high score signals the likelihood for physically abusing a child. All right, got that. But the instrument was developed with a standardization group of people already determined to have abused a child. Okay. Now the test title may be misleading and the examiner, the psychologist may be using the test itself to determine that the person actually abused the child. Right. But he hasn't been, he's not in the same.

John Zervopoulos (28m 37s):
He's not the same kind of person as a standardization sample. So the key takeaway there is, is the test taker, the parent being compared to a different group than what he is or she is all right,

Ryan Kalamaya (28m 54s):
This episode is brought to you by our law firm call Gosia Amy. And I describe our law firm as an innovative and ambitious trial team. The pushes, the boundaries to discover new frontiers in family law, personal injuries in criminal defense in Colorado. We currently have offices in Aspen, Glenwood Springs, Edwards, Denver, and Boulder. If you wanna find out more, visit our website, call.law. Now back to the show. So let's talk about key number three, statistical test reliability. John, tell us about key number three,

John Zervopoulos (29m 32s):
Key number three, statistical test reliability that relates to the consistency of scores. Okay. And reliability is statistical test reliability is different than legal reliability, right? Interestingly enough, DBER in footnote nine makes that distinction, all right. Legal reliability deals with evidentiary reliability, the trustworthiness of the evidence, but simply statistical test reliability refers to the extent to which test scores would be consistent if the test were to be administered to the same person several times. Okay. So that sort of makes sense. A test is not gonna be very reliable if Ryan, you take it today and then three days from now, you come up with different scores and two days later you come up with different scores from there.

John Zervopoulos (30m 22s):
All right. So that's a key issue. One of the issues in terms of reliability is for how long of a period can you expect a test to be, to return the same kind of scores. Now, Dr. Roger Green, who is an eminent authority under an M P I to, and in fact was born in Eastern. Oklahoma grew up in Eastern, Oklahoma. I mean, Eastern Colorado. He advises psychologist not to make long-term predictions from a single administration of the MMPI to, okay. So how, how might that apply? Let's say in a several month child custody evaluation with trial, usually several weeks, if not months beyond the time that the report is filed, the MPI is, and the P is typically administered over, you know, only once.

John Zervopoulos (31m 13s):
All right, how reliable is the P such that the expert is using the P I to inform their opinion, but is testifying several months, if not a year after the MPI was given and the person might score differently on them MPI, all right. At the time of trial than he did during the evaluation and green warns, that the literature's unclear about whether the M P I two profile shifts over time is meaningful changes in the examinee characteristics or the structure of the test or both. So bottom line is looking for consistency.

John Zervopoulos (31m 56s):
And whether just because a test is given at one point, will the person at the time of testimony score the same way, you know, several months or several weeks down the line.

Ryan Kalamaya (32m 10s):
And I can think how this could really impact a P E that's done, you know, maybe on the front range of Colorado where the, sometimes the trial dates or a year or two years off, or someone tries to use psychological testing from a Pere that was done during a divorce. And there's now a post decree dispute, and they try to use the testing two years, three years later, five years later. And, and really what I'm hearing you say, John, and correct me if I'm wrong is that, you know, there's, it's not so fast that there's some argument as to whether or not that testing done is reliable when, you know, depending on the circumstances of when it was administered and when it is brought up.

Ryan Kalamaya (33m 1s):
Right?

John Zervopoulos (33m 2s):
Absolutely. And, but that doesn't necessarily mean that the testing can't be useful for the lawyer or the evaluation can't be useful because for example, the lawyer, you know, you might be able to take the evaluation, say, you know, this was done a year and a half ago, and a lot of it is not relevant now, for example, in the testing, but let's put a pin on the time that the evaluation was completed and see where the, my client was or opposing client was, and then use evidence that you've accumulated in your case since then to prove or disprove what the evaluator came up in the evaluation.

John Zervopoulos (33m 42s):
So that's kind of a strategy to use for old testing and old evaluations as well.

Ryan Kalamaya (33m 48s):
Now test validity is key number four. So John, talk to me about test validity. What does that mean?

John Zervopoulos (33m 55s):
Two issues with test validity, first of all, does the test or scale within a test, accurately measure what it claims to measure? All right, now let's go back just real quickly to the MMPI. It's one test, but it's a test that's devised of several scales. Okay. So each scale also has reliability and validity issues related to it. All right. So validity is, does a test or scale within accurately measure what it, it claims to measure? For example, there's a PTSD scale in the MMPI. The problem is that the scale doesn't accurately measure PTSD, just because a person scores high on it doesn't mean they have PTSD.

John Zervopoulos (34m 43s):
All right. What it measures is symptoms that PTSD actually shares with other diagnoses, right? So that's one element of validity. Does it test or scale actually measure what it claims to measure? The second issue is, is the psychologist using the test in the way that it was intended to be used. And we find this problem with Pires or child custody evaluations, a lot, for example, the mm P I two is not a parenting inventory. Okay. And oftentimes what you find is that the evaluator will be relying overly much on the P I two results estimating that it's sort of like a parenting inventory.

John Zervopoulos (35m 31s):
And I think it's important for the lawyer to separate that out, because if there are problem issues in the MMPI, well, you, you can handle that with other evidence in your case. Right. And make sure that you're separating that out. If that's a problem for you,

Ryan Kalamaya (35m 46s):
John, do you think it, would it be fair to, you know, analogize that like an IQ test, which a lot of people they're familiar with or at least have heard about the IQ test and just because someone is, has a really high IQ and that they are really intelligent, at least according to the IQ score, that doesn't necessarily mean that they're a great parent. There's plenty of people that have very high IQs that could be doing things that are very detrimental as a parent. Is that kind of an, an analogy to, you know, an evaluator kind of seeing an MMPI score index and really kind of equivocating or, you know, relating that to parenting exactly

John Zervopoulos (36m 26s):
What you're looking for. Again, something we mentioned earlier is that the test results inform the evaluator with other information as well, so that the evaluator can make their conclusions in opinions. So we have key four takeaway explore whether the tests that the evaluator use accurately measure what they claim to measure. Remember we talked about the child abuse tests as an example earlier, and determine whether the evaluator used the tests and the test results in the manner for which the test and test results were intended. Okay. So a quick aside, there's a relationship that we need to keep in mind with regard to reliability and validity.

John Zervopoulos (37m 9s):
All right. Think of a target reliability, again, has to deal with consistency. Alright, so you have a target. Let's say that the person is very consistent, but not hitting the bullseye. All right. And another time is consistent, but not hitting the bullseye. They're the test is reliable. Okay. But it may not be valid. Do you see what I'm saying? So those two have to be together a valid test, the bullseye, but all the shots are hitting the bullseye as well. Do you see what I'm saying? So you can't have validity without reliability, but just because the test is reliable, consistent, doesn't mean that it's valid, that it's being used for the purpose that it was designed.

John Zervopoulos (38m 2s):
Hope. That makes sense.

Ryan Kalamaya (38m 4s):
Got it. All right. Yeah. No, it makes sense to me. Hopefully our listeners, they can follow that. And you know, we're, we're finishing up here with our last two keys, so sensitivity and specificity, key number five. So John tell our listeners about sensitivity and specificity.

John Zervopoulos (38m 22s):
Okay. So those are kind of highfalutin words really has to do with the accuracy of the scores. All right. And let's use the example of depression, but as I talk about depression, I want you to think of COVID tests. We're all, all familiar with these home COVID tests. Alright. So sensitivity is how likely is that a person shown by the test results as depressed is really depressed. Okay. If you look at COVID tests, the home tests, supposedly if it shows that you have COVID, then you probably do have COVID all right. But that's only one part of the, of the coin.

John Zervopoulos (39m 2s):
Specificity is the other, how likely is that a person who is not depressed will be identified as not really depressed. Okay. So how likely is it that a person who has a negative COVID test really does not have COVID all right. So we're looking at two sides of this issue because no measurement is perfect. So considered a mental health expert who testifies that a four year old child has been sexually abused because that child has nightmares does not wanna visit the accused parent and frequently touches their own genitals.

John Zervopoulos (39m 43s):
All right. Now, although these characteristics may result from abuse, that sensitivity, right? They also are characteristics of some anxious, young children who have not been abused that's lack of specificity. All right. So you weigh those two issues again, think of the COVID test and realize that the interpretation of the test may differ depending on whether you're looking at it from whether the test shows that you have it, or if you don't have it, it shows it, it doesn't.

Ryan Kalamaya (40m 19s):
So John I've seen in, in these P E files, these, you know, these computer generated interpretive results. So it's all this gobbly G, and that is our final key. The key number six. So talk to me about the computer generated interpretive reports.

John Zervopoulos (40m 38s):
These are important things to keep in mind for any lawyer. All right, these days, the, these tests are scored by computers either directly on online from the psychologist, computer or sent in to the company. And what these generated reports do is they take the scores and generate possibilities in diagnoses and so forth that may come about from these particular scores. And it's important to see that Pearson, the M P I two report has a caution there that is also echoed in other tests.

John Zervopoulos (41m 19s):
And the caution is this, the personality descriptions, inferences and recommendations contained herein need to be verified by other sources of clinical information because individual clients may not have fully matched the prototype. Okay. We've talked about that, right? You need other informations besides just test scores to make these test scores realistic. How do you tell if the per if the examiner the evaluator is using is copying from the computer generated test score, put on our legal writing glasses. All right. And what you'll see oftentimes is you're reading a report and maybe the writer is fairly good writer or other writers are not so good.

John Zervopoulos (42m 6s):
And then you kind get the taste of how the flow of the writing is going. Then suddenly when you get the test results, suddenly it seems like you're reading out of a, a psychology book. All right. That's probably because they're getting that language directly from the computerized report, right. Which means they have the computerized report in their file. So it's important that you challenge the evaluator with a three step process. First ask the evaluator, if he or she ordered the computerized generated test report and asked them to produce it, some may kind of fight you on that saying that it is copyrighted materials and so forth, but that's part of the file.

John Zervopoulos (42m 50s):
That's part of issues of documents at the evaluator relied on, and you should be able to access that from their file. Then ask the evaluator to read you the cautionary statement from the test report. That's usually in any computerized generated report and then have the evaluator go through the report and ask why they adopted certain statements from the report and why they didn't adopt other statements from the report. And you may get some very interesting information about whether the evaluator was picking and choosing, or was really using a test as it was intended to be used.

John Zervopoulos (43m 36s):
And what you'll find out sometimes is that an evaluator will be relying on those computer generat generated reports, reports, and statements to basically say that this is what the test shows therefore, and basically shortcut the process of incorporating the test information with the other information to come to their conclusions.

Ryan Kalamaya (43m 59s):
And John, before we, you know, wrap this up and conclude, you know, key number six, about the interpretive reports that our computers generated, it reminds me of, you know, a case that you and I worked on where, you know, can you tell me, and, and our listeners a little bit about the kind of ethics or the regulations surrounding getting the raw data of these psychological reports and what goes in, can any lawyer just ask for the raw answers to these psychological reports? And if not, why? Why not?

John Zervopoulos (44m 31s):
You know, it's almost state by state the American psychological Association's ethics code basically has said that the raw data can be disclosed, but not the test questions. All right. The wave testing used to be before Daubert in, in these sorts of case where experts had to disclose everything in their file. The questions used to be on the same page as the answer. You have the question true false, or 1, 2, 3, 4, 5, all right. But since then the test publishers, because of notions of copyright and so forth, have separated out the questions from the test answers.

John Zervopoulos (45m 18s):
So the, the psychologists should be able to, according to APA, disclose the test answers without having to disclose the test questions. Okay. And that's good. All right. Included. He should, the psychologist should be able to disclose the computer computerized generated report. However, in many states they have the old style deal where raw data is included with the notions of the test questions, and they can, those materials can only be submitted to another psychologist or someone trained in testing. So find out what happens in your particular state and then kind of work, work around that way.

Ryan Kalamaya (46m 4s):
And you know, the final question for you is when you have two different evaluators, let's say there's AE and then a supplemental and the supplemental period does, you know, ink blot for psychological testing was the roar shock. Talk to me about when psychologists use different tests and there being arguments about which one is, you know, better than another,

John Zervopoulos (46m 31s):
Well, most rely on the NPI to, in the MCMI. All right. And or the PAI, the personality assessment inventory, those are kind of characterized as objective tests because they have scales they're compared statistically with standardized samples and so forth. The Rosh shock test or familiar with the ink plot test is more what they call subjective projective kind of tests. And the notion is that the person is projecting their own inner issues onto the ink blocks as they describe the ink blocks. And the psychologist kind of describes from there what the issues are and the problems that the person is having from how they're describing those in blocks.

John Zervopoulos (47m 14s):
All right, there's an interpreter system for the RO shock that some psychologists use that has some similarities to the MMPI and the MCMI. But in my view, what you would really need to do is the, the lawyer in my view, what the lawyer needs to do is to insist on what the MMPI is showing and ask the expert to justify why they're not using what's more traditionally objective tests. And then look at how the evaluator is using those objective tests. Usually they're trying to use it to confirm and justify their conclusions from other areas of the evaluation.

John Zervopoulos (48m 3s):
And I would just have them compare the MPI with the Roshak test in terms of statistical stability and the use of it in child custody evaluations. Those are the sort of tests that are more useful in clinical situations than they would be in forensic situations where a different metric is used to justify using these assessment instruments.

Ryan Kalamaya (48m 27s):
Got it. So it could be used to augment the existing testing or the testing that was done by the first evaluator. And that doesn't necessarily mean that it goes back to some of your other key points about, you know, them, the evaluator, and to what extent they really rely on it. So, John, for us to just kind of wrap

John Zervopoulos (48m 48s):
Up, but Ryan, you could still use the other keys to evaluate the, their use of the RO shock as well. Right? You can ask about the reliability. You can ask about the validity issues that we discuss. You could ask about the standardization sample, if any was used. And if they just say, well, this is a subjective sort of thing, and I'm using my own clinical judgment without comparing to other people who have similar observations of the ink lot. That's a problem. And so you can use these keys as ways of kind of cross-examining or unpacking the evaluator use of the RO shop data.

Ryan Kalamaya (49m 29s):
Got it. Well, to wrap things up, John, what are some conclusion, you know, some, some final thoughts or takeaways you provided a fair amount there and had a key takeaway for each key point, but what are the overall takeaways for an attorney listening to this? If they've gotten this far or a party going through in a, a parenting evaluation on psychological testing?

John Zervopoulos (49m 52s):
Well, there are a few, first of all, keep in mind that psychological tests are not akin to medical. X-rays just have that in your mind and your way down the road already. All right. Secondly, view interpretation of test results. As context related, remember our three-legged foot stool test results have to be interpreted in context. The other information from interviews, the collateral material, and also the case facts. Another issue is you don't have to get deep into the weeds to question experts about testing, but know where you are in the weeds, stay with your keys.

John Zervopoulos (50m 36s):
Don't let the expert get into describing statistics and so forth in ways that take you off what your goal is, follow the keys and get what you need to from the expert in that way. And finally, although testing can provide useful information. It's not the only information. And if the testing is not helpful to you really lean into the interviews and the collateral material in the case facts and your case story to flesh out what you need to get from the experts. Remember not everyone qualified to give, to do conduct a P E does psychological testing, so it's not necessary.

John Zervopoulos (51m 19s):
And if it works against you lean hard into the other elements of an evaluation to get your point across, to get your story across for your case.

Ryan Kalamaya (51m 29s):
Well, John, as Zoes, whenever I have a conversation with you, I'm just, it's, it's like a fire drinking through a fire hydrant because it, it just is learning so much. And obviously you're knowledgeable on so many different areas involving, you know, custody evaluations and PEs and CFIs. So thanks again for joining us. If people wanna learn more, they'll have a link to your website in the show notes, but just for people listening where what's the best way to reach out to you, if they wanna find out more, whether they're an attorney or there's someone going through an evaluation, or they've gotten an adverse or helpful opinion in an evaluation, and they want a consultant and expert like you to really help you know, them understand the strengths and weaknesses of a particular evaluation.

John Zervopoulos (52m 17s):
So what I, I would suggest is that is that they had their attorney call me or email me and my go to my website@wwwdotpsychologylawpartners.com, www psychology law partners.com. And there's all the information about my services, my background, and what I can provide in terms of my expertise to lawyers. And I'd be more than happy to exchange emails or talk with you about your case,

Ryan Kalamaya (52m 48s):
John. Thank you again. Thank you until next time. This is Ryan. Callam hiring off for divorce altitude. Hey everyone. This is Ryan again. Thank you for joining us on divorce at altitude. If you found our tips, insight or discussion, helpful, please tell a friend about this podcast for show notes, additional resources or links mentioned on today's episode, visit divorce, altitude.com. Follow us on apple podcasts, Spotify, or wherever you listen in. Many of our episodes are also posted on YouTube. You can also find Amy and me@callam.law or 9, 7 0 3 1 5 2 3 6 5.

Ryan Kalamaya (53m 28s):
That's.