Reliability

Reliability is the extent to which an experiment, test, or any measuring procedure yields the same result on repeated trials. Without the agreement of independent observers able to replicate research procedures, or the ability to use research tools and procedures that yield consistent measurements, researchers would be unable to satisfactorily draw conclusions, formulate theories, or make claims about the generalizability of their research. In addition to its important role in research, reliability is critical for many parts of our lives, including manufacturing, medicine, and sports.

According to the Dictionary:

"Yielding the same or compatible results in different clinical experiments or statistical trials"

The idea behind reliability is that any significant results must be more than a one-off finding and be inherently repeatable.

Other researchers must be able to perform exactly the same experiment, under the same conditions and generate the same results. This will reinforce the findings and ensure that the wider scientific community will accept the hypothesis. Without this replication of statistically significant results, the experiment and research have not fulfilled all of the requirements of testability. This prerequisite is essential to a hypothesis establishing itself as an accepted scientific truth.
For example, if you are performing a time critical experiment, you will be using some type of stopwatch. Generally, it is reasonable to assume that the instruments are reliable and will keep true and accurate time. However, diligent scientists take measurements many times, to minimize the chances of malfunction and maintain validity and reliability.
At the other extreme, any experiment that uses human judgment is always going to come under question. This means that such experiments are more difficult to repeat and are inherently less reliable. Reliability is a necessary ingredient for determining the overall validity of a scientific experiment and enhancing the strength of the results.

Reliability and science

Reliability is something that every scientist, especially in social sciences and biology, must be aware of. In science, the definition is the same, but needs a much narrower and unequivocal definition.
Another way of looking at this is as maximizing the inherent repeatability or consistency in an experiment. For maintaining reliability internally, a researcher will use as many repeat sample groups as possible, to reduce the chance of an abnormal sample group skewing the results. If you use three replicate samples for each manipulation, and one generates completely different results from the others, then there may be something wrong with the experiment.

1. For many experiments, results follow a ‘normal distribution’ and there is always a chance that your sample group produces results lying at one of the extremes. Using multiple sample groups will smooth out these extremes and generate a more accurate spread of results.
2. If your results continue to be wildly different, then there is likely to be something very wrong with your design; it is unreliable.

Reliability and cold fusion

Reliability is also extremely important externally, and another researcher should be able to perform exactly the same experiment, with similar equipment, under similar conditions, and achieve exactly the same results. If they cannot, then the design is unreliable.

A good example of a failure to apply the definition of reliability correctly is provided by the cold fusion case, of 1989. Fleischmann and Pons announced to the world that they had managed to generate heat at normal temperatures, instead of the huge and expensive tori used in most research into nuclear fusion.
This announcement shook the world, but researchers in many other institutions across the world attempted to replicate the experiment, with no success. Whether the researchers lied, or genuinely made a mistake is unclear, but their results were clearly unreliable.

Reliability and Statistics

Physical scientists expect to obtain exactly the same results every single time, due to the relative predictability of the physical realms. If you are a nuclear physicist or an inorganic chemist, repeat experiments should give exactly the same results, time after time.

Ecologists and social scientists, on the other hand, understand fully that achieving exactly the same results is an exercise in futility. Research in these disciplines incorporates random factors and natural fluctuations and, whilst any experimental design must attempt to eliminate confounding variables and natural variations, there will always be some disparities.
The key to performing a good experiment is to make sure that your results are as reliable as is possible; if anybody repeats the experiment, powerful statistical tests will be able to compare the results and the scientist can make a solid estimate of statistical reliability.

Testing reliability for social sciences and Education

In the social sciences, testing reliability is a matter of comparing two different versions of the instrument and ensuring that they are similar. When we talk about instruments, it does not necessarily mean a physical instrument, such as a mass-spectrometer or a pH-testing strip.

An educational test, questionnaire, or assigning quantitative scores to behavior are also instruments, of a non-physical sort. Measuring the reliability of instruments occurs in different ways.

Types of Reliability

There are four general classes of reliability estimates, each of which estimates reliability in a different way. They are:

• Inter-Rater or Inter-Observer Reliability
  Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon.
• Test-Retest Reliability
  Used to assess the consistency of a measure from one time to another.
• Parallel-Forms Reliability
  Used to assess the consistency of the results of two tests constructed in the same way from the same content domain.
• Internal Consistency Reliability
  Used to assess the consistency of results across items within a test.

Inter-Rater or Inter-Observer Reliability

Whenever you use humans as a part of your measurement procedure, you have to worry about whether the results you get are reliable or consistent. People are notorious for their inconsistency. We are easily distractible. We get tired of doing repetitive tasks. We daydream. We misinterpret.
So how do we determine whether two observers are being consistent in their observations? You probably should establish inter-rater reliability outside of the context of the measurement in your study. After all, if you use data from your study to establish reliability, and you find that reliability is low, you're kind of stuck. Probably it's best to do this as a side study or pilot study. And, if your study goes on for a long time, you may want to reestablish inter-rater reliability from time to time to assure that your raters aren't changing.
There are two major ways to actually estimate inter-rater reliability. If your measurement consists of categories -- the raters are checking off which category each observation falls in -- you can calculate the percent of agreement between the raters. For instance, let's say you had 100 observations that were being rated by two raters. For each observation, the rater could check one of three categories. Imagine that on 86 of the 100 observations the raters checked the same category. In this case, the percent of agreement would be 86%. OK, it's a crude measure, but it does give an idea of how much agreement exists, and it works no matter how many categories are used for each observation.
The other major way to estimate inter-rater reliability is appropriate when the measure is a continuous one. There, all you need to do is calculate the correlation between the ratings of the two observers. For instance, they might be rating the overall level of activity in a classroom on a 1-to-7 scale. You could have them give their rating at regular time intervals (e.g., every 30 seconds). The correlation between these ratings would give you an estimate of the reliability or consistency between the raters.
You might think of this type of reliability as "calibrating" the observers. There are other things you could do to encourage reliability between observers, even if you don't estimate it. For instance, I used to work in a psychiatric unit where every morning a nurse had to do a ten-item rating of each patient on the unit. Of course, we couldn't count on the same nurse being present every day, so we had to find a way to assure that any of the nurses would give comparable ratings. The way we did it was to hold weekly "calibration" meetings where we would have all of the nurses ratings for several patients and discuss why they chose the specific values they did. If there were disagreements, the nurses would discuss them and attempt to come up with rules for deciding when they would give a "3" or a "4" for a rating on a specific item. Although this was not an estimate of reliability, it probably went a long way toward improving the reliability between raters.

Qualitative assessments and interrater reliability

Any qualitative assessment using two or more researchers must establish interrater reliability to ensure that the results generated will be useful.

One good example is Bandura’s Bobo Doll experiment, which used a scale to rate the levels of displayed aggression in young children. Apart from extensive pre-testing, the observers constantly compared and calibrated their ratings, adjusting their scales to ensure that they were as similar as possible.

Instrument reliability

Instrument reliability is a way of ensuring that any instrument used for measuring experimental variables gives the same results every time.

Martyn Shuttleworth (2009)

In the physical sciences, the term is self-explanatory, and it is a matter of making sure that every piece of hardware, from a mass spectrometer to a set of weighing scales, is properly calibrated.

Instruments in research

As an example, a researcher will always test the instrument reliability of weighing scales with a set of calibration weights, ensuring that the results given are within an acceptable margin of error.

Some of the highly accurate balances can give false results if they are not placed upon a completely level surface, so this calibration process is the best way to avoid this.
In the non-physical sciences, the definition of an instrument is much broader, encompassing everything from a set of survey questions to an intelligence test. A survey to measure reading ability in children must produce reliable and consistent results if it is to be taken seriously.
Political opinion polls, on the other hand, are notorious for producing inaccurate results and delivering a near unworkable margin of error.
In the physical sciences, it is possible to isolate a measuring instrument from external factors, such as environmental conditions and temporal factors. In the social sciences, this is much more difficult, so any instrument must be tested with a reasonable range of reliability.

Test of stability

Any test of instrument reliability must test how stable the test is over time, ensuring that the same test performed upon the same individual gives exactly the same results.

The test-retest method is one way of ensuring that any instrument is stable over time.
Of course, there is no such thing as perfection and there will be always be some disparity and potential for regression, so statistical methods are used to determine whether the stability of the instrument is within acceptable limits.

Test of equivalence

Testing equivalence involves ensuring that a test administered to two people, or similar tests administered at the same time give similar results.

Split-testing is one way of ensuring this, especially in tests or observations where the results are expected to change over time. In a school exam, for example, the same test upon the same subjects will generally result in better results the second time around, so testing stability is not practical.
Checking that two researchers observe similar results also falls within the remit of the test of equivalence.

Test-retest reliability

The test-retest reliability method is one of the simplest ways of testing the stability and reliability of an instrument (of a research) over time.

Martyn Shuttleworth (2009)

For example, in a research, if a group of students takes a test, you would expect them to show very similar results if they take the same test a few weeks later. This definition relies upon there being no confounding factor during the intervening time interval.
Instruments in research such as IQ tests and surveys are prime candidates for test-retest methodology, because there is little chance of people experiencing a sudden jump in IQ or suddenly changing their opinions.
On the other hand, research data regarding educational tests are often not suitable, because students will learn much more information over the intervening period and show better results in the second test.
We estimate test-retest reliability when we administer the same test to the same sample on two different occasions. This approach assumes that there is no substantial change in the construct being measured between the two occasions. The amount of time allowed between measures is critical. We know that if we measure the same thing twice that the correlation between the two observations will depend in part by how much time elapses between the two measurement occasions. The shorter the time gap, the higher the correlation; the longer the time gap, the lower the correlation. This is because the two observations are related over time -- the closer in time we get the more similar the factors that contribute to error. Since this correlation is the test-retest estimate of reliability, you can obtain considerably different estimates depending on the interval.

Parallel-Forms Reliability

In parallel forms reliability you first have to create two parallel forms. One way to accomplish this is to create a large set of questions that address the same construct and then randomly divide the questions into two sets. You administer both instruments to the same sample of people. The correlation between the two parallel forms is the estimate of reliability. One major problem with this approach is that you have to be able to generate lots of items that reflect the same construct. This is often no easy feat. Furthermore, this approach makes the assumption that the randomly divided halves are parallel or equivalent. Even by chance this will sometimes not be the case. The parallel forms approach is very similar to the split-half reliability described below. The major difference is that parallel forms are constructed so that the two forms can be used independent of each other and considered equivalent measures. For instance, we might be concerned about a testing threat to internal validity. If we use Form A for the pretest and Form B for the posttest, we minimize that problem. it would even be better if we randomly assign individuals to receive Form A or B on the pretest and then switch them on the posttest. With split-half reliability we have an instrument that we wish to use as a single measurement instrument and only develop randomly split halves for purposes of estimating reliability.

Internal Consistency Reliability

In internal consistency reliability estimation we use our single measurement instrument administered to a group of people on one occasion to estimate reliability. In effect we judge the reliability of the instrument by estimating how well the items that reflect the same construct yield similar results. We are looking at how consistent the results are for different items for the same construct within the measure. There are a wide variety of internal consistency measures that can be used.

Internal consistency reliability defines the consistency of the results delivered in a test, ensuring that the various items measuring the different constructs deliver consistent scores.

Martyn Shuttleworth (2009)

For example, an English test is divided into vocabulary, spelling, punctuation and grammar. The internal consistency reliability test provides a measure that each of these particular aptitudes is measured correctly and reliably.
One way of testing this is by using a test–retest method, where the same test is administered some after the initial test and the results compared.
However, this creates some problems and so many researchers prefer to measure internal consistency by including two versions of the same instrument within the same test. Our example of the English test might include two very similar questions about comma use, two about spelling and so on.
The basic principle is that the student should give the same answer to both – if they do not know how to use commas, they will get both questions wrong. A few nifty statistical manipulations will give the internal consistency reliability and allow the researcher to evaluate the reliability of the test.
There are three main techniques for measuring the internal consistency reliability, depending upon the degree, complexity and scope of the test.
They all check that the results and constructs measured by a test are correct, and the exact type used is dictated by subject, size of the data set and resources.

Split-Halves test

The split halves test for internal consistency reliability is the easiest type, and involves dividing a test into two halves.

For example, a questionnaire to measure extroversion could be divided into odd and even questions. The results from both halves are statistically analyzed, and if there is weak correlation between the two, then there is a reliability problem with the test.

The split halves test gives a measurement of in between zero and one, with one meaning a perfect correlation.

The division of the question into two sets must be random. Split halves testing was a popular way to measure reliability, because of its simplicity and speed.

However, in an age where computers can take over the laborious number crunching, scientists tend to use much more powerful tests.
In split-half reliability we randomly divide all items that purport to measure the same construct into two sets. We administer the entire instrument to a sample of people and calculate the total score for each randomly divided half. the split-half reliability estimate, as shown in the figure, is simply the correlation between these two total scores. In the example it is .87.

Cronbach’s Alpha test

The Cronbach's Alpha test not only averages the correlation between every possible combination of split halves, but it allows multi-level responses.

For example, a series of questions might ask the subjects to rate their response between one and five. Cronbach's Alpha gives a score of between zero and one, with 0.7 generally accepted as a sign of acceptable reliability.
The test also takes into account both the size of the sample and the number of potential responses. A 40-question test with possible ratings of 1 – 5 is seen as having more accuracy than a ten-question test with three possible levels of response.
Of course, even with Cronbach's clever methodology, which makes calculation much simpler than crunching through every possible permutation, this is still a test best left to computers and statistics spreadsheet programmes.

Kudar-Richardson test

The Kudar Richardson test for internal consistency reliability is a more advanced, and slightly more complex, version of the split halves test.

In this version, the test works out the average correlation for all the possible split half combinations in a test. The Kudar Richardson test also generates a correlation of between zero and one, with a more accurate result than the split halves test. The weakness of this approach, as with split-halves, is that the answer for each question must be a simple right or wrong answer, zero or one.
For multi-scale responses, sophisticated techniques are needed to measure internal consistency reliability.

Validity in Research

"Any research can be affected by different kinds of factors which, while extraneous to the concerns of the research, can invalidate the findings"

(Seliger & Shohamy 1989, 95).

Validity refers to the degree to which a study accurately reflects or assesses the specific concept that the researcher is attempting to measure. Validity is concerned with the study's success at measuring what the researchers set out to measure.

Researchers should be concerned with both external and internal validity. External validity refers to the extent to which the results of a study are generalizable or transferable.

Validity encompasses the entire experimental concept and establishes whether the results obtained meet all of the requirements of the scientific research method.

Types of validity

Scholars discuss several types of internal validity.

• Face Validity
• Criterion Related Validity
• Construct Validity
• Content Validity

Face Validity

Face validity is a measure of how representative a research project is ‘at face value,’ and whether it appears to be a good project.
Face validity is concerned with how a measure or procedure appears. Does it seem like a reasonable way to gain the information the researchers are attempting to obtain? Does it seem well designed? Does it seem as though it will work reliably? Unlike content validity, face validity does not depend on established theories for support
This is the least scientific method of validity as it is not quantified using statistical methods.  This is not validity in a technical sense of the term.  It is concerned with whether it seems like we measure what we claim.  Here we look at how valid a measure appears on the surface and make subjective judgments based off of that.  For example, if you give a survey that appears to be valid to the respondent and the questions are selected because they look valid to the administer. The administer asks a group of random people, untrained observers, if the questions appear valid to them.  In research it’s never sufficient to rely on face judgments alone and more quantifiable methods of validity are necessary in order to draw acceptable conclusions.  There are many instruments of measurement to consider so face validity is useful in cases where you need to distinguish one approach over another.  Face validity should never be trusted on its own merits.

Criterion Related Validity

Criterion related validity, also referred to as instrumental validity, is used to demonstrate the accuracy of a measure or procedure by comparing it with another measure or procedure which has been demonstrated to be valid.
For example, imagine a hands-on driving test has been shown to be an accurate test of driving skills. By comparing the scores on the written driving test with the scores from the hands-on driving test, the written test can be validated by using a criterion related strategy in which the hands-on driving test is compared to the written test.
The accuracy of a measure is demonstrated by comparing it with a measure that has been demonstrated to be valid. In other words, correlation with other measures that have known validity. For this to work you must know that the criterion has been measured well.  And be aware that appropriate criteria do not always exist.  What you are doing is checking the performance of your operationalization against criteria.  The criteria you use as a standard of judgment accounts for the different approaches you would use:

• Predictive Validity - operationalization’s ability to predict what it is theoretically able to predict.  The extent to which a measure predicts expected outcomes.
• Concurrent Validity - operationalization’s ability to distinguish between groups it theoretically should be able to.  This is where a test correlates well with a measure that has been previously validated.

Construct Validity

Construct validity seeks agreement between a theoretical concept and a specific measuring device or procedure. For example, a researcher inventing a new IQ test might spend a great deal of time attempting to "define" intelligence in order to reach an acceptable level of construct validity.
To understand whether a piece of research has construct validity, three steps should be followed.

·        First, the theoretical relationships must be specified.

·        Second, the empirical relationships between the measures of the concepts must be examined.

·        Third, the empirical evidence must be interpreted in terms of how it clarifies the construct validity of the particular measure being tested

Construct validity defines how well a test or experiment measures up to its claims. A test designed to measure depression must only measure that particular construct, not closely related ideals such as anxiety or stress.

• Convergent validity tests that constructs that are expected to be related are, in fact, related.
• Discriminate validity tests that constructs that should have no relationship do, in fact, not have any relationship. (also referred to as divergent validity)

A construct represents a collection of behaviors that are associated in a meaningful way to create an image or an idea invented for a research purpose.  Depression is a construct that represents a personality trait which manifests itself in behaviors such as over sleeping, loss of appetite, difficulty concentrating, etc.  The existence of a construct is manifest by observing the collection of related indicators.  Any one sign may be associated with several constructs.  A person with difficulty concentrating may have A.D.D. but not depression.  Construct validity is the degree to which inferences can be made from operationalizations (connecting concepts to observations) in your study to the constructs on which those operationalizations are based.  To establish construct validity you must first provide evidence that your data supports the theoretical structure.  You must also show that you control the operationalization of the construct, in other words, show that your theory has some correspondence with reality.

Content Validity

Content validity is the estimate of how much a measure represents every single element of a construct.
Content validity is illustrated using the following examples: Researchers aim to study mathematical learning and create a survey to test for mathematical skill. If these researchers only tested for multiplication and then drew conclusions from that survey, their study would not show content validity because it excludes other mathematical functions. Although the establishment of content validity for placement-type exams seems relatively straight-forward, the process becomes more complex as it moves into the more abstract domain of socio-cultural studies. For example, a researcher needing to measure an attitude like self-esteem must decide what constitutes a relevant domain of content for that attitude. For socio-cultural studies, content validity forces the researchers to define the very domains they are attempting to study.
This is also a subjective measure but unlike face validity we ask whether the content of a measure covers the full domain of the content. If a researcher wanted to measure introversion they would have to first decide what constitutes a relevant domain of content for that trait.  This is considered a subjective form of measurement because it still relies on people’s perception for measuring constructs that would otherwise be difficult to measure.   Where it distinguishes itself is through its use of experts in the field or individuals belonging to a target population.  This study can be made more objective through the use of rigorous statistical tests.  For example you could have a content validity study that informs researchers how items used in a survey represent their content domain, how clear they are, and the extent to which they maintain the theoretical factor structure assessed by the factor analysis.
Internal Validity
Internal validity refers to the rigor with which the study was conducted (e.g., the study's design, the care taken to conduct measurements, and decisions concerning what was and wasn't measured). This refers to the extent to which the independent variable can accurately be stated to produce the observed effect.  If the effect of the dependent variable is only due to the independent variable(s) then internal validity is achieved. This is the degree to which a result can be manipulated.

Internal validity is a measure which ensures that a researcher’s experiment design closely follows the principle of cause and effect.

“Could there be an alternative cause, or causes, that explain my observations and results?”

Internal validity dictates how an experimental design is structured and encompasses all of the steps of the scientific research method.
Even if your results are great, sloppy and inconsistent design will compromise your integrity in the eyes of the scientific community. Internal validity and reliability are at the core of any experimental design.
External Validity
The extent to which the designers of a study have taken into account alternative explanations for any causal relationships they explore (Huitt, 1998).This refers to the extent to which the results of a study can be generalized beyond the sample. Which is to say that you can apply your findings to other people and settings?   Think of this as the degree to which a result can be generalized.
External validity is usually split into two distinct types, population validity and ecological validity and they are both essential elements in judging the strength of an experimental design.
External validity is the process of examining the results and questioning whether there are any other possible causal relationships.

In 1966, Campbell and Stanley proposed the commonly accepted definition of external validity.

“External validity asks the question of generalizability: To what populations, settings, treatment variables and measurement variables can this effect be generalized?”

External validity is one the most difficult of the validity types to achieve, and is at the foundation of every good experimental design.

By Martyn Shuttleworth (2009)

Many scientific disciplines, especially the social sciences, face a long battle to prove that their findings represent the wider population in real world situations.
The main criteria of external validity is the process of generalization, and whether results obtained from a small sample group, often in laboratory surroundings, can be extended to make predictions about the entire population.

Reliability & Validity

Reliability and validity are often confused, but the terms actually describe two completely different concepts, although they are often closely inter-related. This distinct difference is best summed up with an example:

A researcher devises a new test that measures IQ more quickly than the standard IQ test:

• If the new test delivers scores for a candidate of 87, 65, 143 and 102, then the test is not reliable or valid, and it is fatally flawed.
• If the test consistently delivers a score of 100 when checked, but the candidates real IQ is 120, then the test is reliable, but not valid.
• If the researcher’s test delivers a consistent score of 118, then that is pretty close, and the test can be considered both valid and reliable.

Reliability is an essential component of validity but, on its own, is not a sufficient measure of validity. A test can be reliable but not valid, whereas a test cannot be valid yet unreliable.
Reliability, in simple terms, describes the repeatability and consistency of a test. Validity defines the strength of the final results and whether they can be regarded as accurately describing the real world.

We often think of reliability and validity as separate ideas but, in fact, they're related to each other. Here, I want to show you two ways you can think about their relationship.
One of my favorite metaphors for the relationship between reliability is that of the target. Think of the center of the target as the concept that you are trying to measure. Imagine that for each person you are measuring, you are taking a shot at the target. If you measure the concept perfectly for a person, you are hitting the center of the target. If you don't, you are missing the center. The more you are off for that person, the further you are from the center.

The figure above shows four possible situations. In the first one, you are hitting the target consistently, but you are missing the center of the target. That is, you are consistently and systematically measuring the wrong value for all respondents. This measure is reliable, but no valid (that is, it's consistent but wrong). The second shows hits that are randomly spread across the target. You seldom hit the center of the target but, on average, you are getting the right answer for the group (but not very well for individuals). In this case, you get a valid group estimate, but you are inconsistent. Here, you can clearly see that reliability is directly related to the variability of your measure. The third scenario shows a case where your hits are spread across the target and you are consistently missing the center. Your measure in this case is neither reliable nor valid. Finally, we see the "Robin Hood" scenario -- you consistently hit the center of the target. Your measure is both reliable and valid (I bet you never thought of Robin Hood in those terms before).
Another way we can think about the relationship between reliability and validity is shown in the figure below. Here, we set up a 2x2 table. The columns of the table indicate whether you are trying to measure the same or different concepts. The rows show whether you are using the same or different methods of measurement. Imagine that we have two concepts we would like to measure, student verbal and math ability. Furthermore, imagine that we can measure each of these in two ways. First, we can use a written, paper-and-pencil exam (very much like the SAT or GRE exams). Second, we can ask the student's classroom teacher to give us a rating of the student's ability based on their own classroom observation.

The first cell on the upper left shows the comparison of the verbal written test score with the verbal written test score. But how can we compare the same measure with itself? We could do this by estimating the reliability of the written test through a test-retest correlation, parallel forms, or an internal consistency measure. What we are estimating in this cell is the reliability of the measure.
The cell on the lower left shows a comparison of the verbal written measure with the verbal teacher observation rating. Because we are trying to measure the same concept, we are looking at convergent validity.
The cell on the upper right shows the comparison of the verbal written exam with the math written exam. Here, we are comparing two different concepts (verbal versus math) and so we would expect the relationship to be lower than a comparison of the same concept with itself (e.g., verbal versus verbal or math versus math). Thus, we are trying to discriminate between two concepts and we would consider this discriminate validity.
Finally, we have the cell on the lower right. Here, we are comparing the verbal written exam with the math teacher observation rating. Like the cell on the upper right, we are also trying to compare two different concepts (verbal versus math) and so this is a discriminate validity estimate. But here, we are also trying to compare two different methods of measurement (written exam versus teacher observation rating). So, we'll call this very discriminate to indicate that we would expect the relationship in this cell to be even lower than in the one above it.
The four cells incorporate the different values that we examine in the multitrait-multimethod approach to estimating construct validity.
When we look at reliability and validity in this way, we see that, rather than being distinct, they actually form a continuum. On one end is the situation where the concepts and methods of measurement are the same (reliability) and on the other is the situation where concepts and methods of measurement are different.

 References

      Educational Research by L. R. Gay (Fifth edition)

      Measurement and evaluation in psychology and education by R. M. Thorndike (fifth edition)

      http://www.socialresearchmethods.net/kb/introval.php

      http://www.experiment-resources.com/research-process.html

      http://www.socialresearchmethods.net/kb/reliable.php

      http://writing.colostate.edu/guides/research/relval/pop2b.cfm

arranged by Sadaf Naz

Steps of Evaluation

Measurement and its kind

Code of Professional Responsibilities in Educational Measurement

Prepared by the NCME Ad Hoc Committee on the Development of a Code of Ethics:
Cynthia B. Schmeiser, ACT--Chair
Kurt F. Geisinger, State University of New York
Sharon Johnson-Lewis, Detroit Public Schools
Edward D. Roeber, Council of Chief State School Officers
William D. Schafer, University of Maryland copyright 1995 National Council on Measurement in Education
Any portion of this Code may be reproduced and disseminated for educational purposes. As an organization dedicated to the improvement of measurement and evaluation practice in education, the National Council on Measurement in Education (NCME) has adopted this Code to promote professionally responsible practice in educational measurement. Professionally responsible practice is conduct that arises from either the professional standards of the field, general ethical principles, or both.
The purpose of the Code of Professional Responsibilities in Educational Measurement, hereinafter referred to as the Code, is to guide the conduct of NCME members who are involved in any tinge of assessment activity in education. NCME is also providing this Code as a public service for all individuals who are engaged in educational assessment activities in the hope that these activities will be conducted in a professionally responsible manner. Persons who engage in these activities include local educators such as classroom teachers, principals, and superintendents; professionals such as school psychologists and counselors; state and national technical, legislative, and policy staff in education; staff of research, evaluation, and testing organizations; providers of test preparation services; college and university faculty and administrators; and professionals in business and industry who design and implement educational and training programs.
This Code applies to any type of assessment that occurs as part of the educational process, including formal and informal, traditional and alternative techniques for gathering information used in making educational decisions at all levels. These techniques include, but are not limited to, large-scale assessments at the school, district, state, national, and international levels; standardized tests; observational measures; teacher conducted assessments; assessment support materials; and other achievement, aptitude, interest, and personality measures used in and for education.
Although NCME is promulgating this Code for its members, it strongly encourages other organizations and individuals who engage in educational assessment activities to endorse and abide by the responsibilities relevant to their professions. Because the Code pertains only to uses of assessment in education, it is recognized that uses of assessments outside of educational contexts, such as for employment, certification, or licensure, may involve additional professional responsibilities beyond those detailed in this Code.
The Code is intended to serve an educational function: to inform and remind those involved in educational assessment of their obligations to uphold the integrity of the manner in which assessments are developed, used, evaluated, and marketed. Moreover, it is expected that the Code will stimulate thoughtful discussion of what constitutes professionally responsible assessment practice at all levels in education.
The Code enumerates professional responsibilities in eight major areas of assessment activity. Specifically, the Code presents the professional responsibilities of those who:
1) Develop Assessments
2) Market and Sell Assessments
3) Select Assessments
4) Administer Assessments
5) Score Assessments
6) Interpret, Use, and Communicate Assessment Results
7) Educate About Assessment
8) Evaluate Programs and Conduct Research on Assessments
Although the organization of the Code is based on the differentiation of these activities, they are viewed as highly interrelated, and those who use this Code are urged to consider the Code in its entirety. The index following this Code provides a listing of some of the critical interest topics within educational measurement that focus on one or more of the assessment activities.

General Responsibilities

The professional responsibilities promulgated in this Code in eight major areas of assessment activity are based on expectations that NCME members involved in educational assessment will: 1) protect the health and safety of all examinees;
2) be knowledgeable about, and behave in compliance with, state and federal laws relevant to the conduct of professional activities;
3) maintain and improve their professional competence in educational assessment;
4) provide assessment services only in areas of their competence and experience, affording full disclosure of their professional qualifications;
5) promote the understanding of sound assessment practices in education;
6) adhere to the highest standards of conduct and promote professionally responsible conduct within educational institutions and agencies that provide educational services; and
7) perform all professional responsibilities with honesty, integrity, due care, and fairness.
Responsible professional practice includes being informed about and acting in accordance with the Code of Fair Testing Practices in Education (Joint Committee on Testing Practices, 1988), the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, National Council on Measurement in Education, 1985), or subsequent revisions, as well as all applicable state and federal laws that may govern the development, administration, and use of assessments. Both the Standards for Educational and Psychological Testing and the Code of Fair Testing Practices in Education are intended to establish criteria for judging the technical adequacy of tests and the appropriate uses of tests and test results. The purpose of this Code is to describe the professional responsibilities of those individuals who are engaged in assessment activities. As would be expected, there is a strong relationship between professionally responsible practice and sound educational assessments, and this Code is intended to be consistent with the relevant parts of both of these documents.
It is not the intention of NCME to enforce the professional responsibilities stated in the Code or to investigate allegations of violations to the Code.
Since the Code provides a frame of reference for the evaluation of the appropriateness of behavior, NCME recognizes that the Code may be used in legal or other similar proceedings.

Section 1:
Responsibilities of Those Who Develop Assessment Products and Services

Those who develop assessment products and services, such as classroom teachers and other assessment specialists, have a professional responsibility to strive to produce assessments that are of the highest quality. Persons who develop assessments have a professional responsibility to:

1.1

Ensure that assessment products and services are developed to meet applicable professional, technical, and legal standards.

1.2

Develop assessment products and services that are as free as possible from bias due to characteristics irrelevant to the construct being measured, such as gender, ethnicity, race, socioeconomic status, disability, religion, age, or national origin.

1.3

Plan accommodations for groups of test takers with disabilities and other special needs when developing assessments.

1.4

Disclose to appropriate parties any actual or potential conflicts of interest that might influence the developers' judgment or performance.

1.5

Use copyrighted materials in assessment products and services in accordance with state and federal law.

1.6

Make information available to appropriate persons about the steps taken to develop and score the assessment, including up to date information used to support the reliability, validity, scoring and reporting processes, and other relevant characteristics of the assessment.

1.7

Protect the rights to privacy of those who are assessed as part of the assessment development process.

1.8

Caution users, in clear and prominent language, against the most likely misinterpretations and misuses of data that arise out of the assessment development process.

1.9

Avoid false or unsubstantiated claims in test preparation and program support materials and services about an assessment or its use and interpretation.

1.10

Correct any substantive inaccuracies in assessments or their support materials as soon as feasible.

1.11

Develop score reports and support materials that promote the understanding of assessment results.

Section 2:
Responsibilities of Those Who Market and Sell Assessment Products and Services

The marketing of assessment products and services, such as tests and other instruments, scoring services, test preparation services, consulting, and test interpretive services, should be based on information that is accurate, complete, and relevant to those considering their use. Persons who market and sell assessment products and services have a professional responsibility to:

2.1

Provide accurate information to potential purchasers about assessment products and services and their recommended uses and limitations.

2.2

Not knowingly withhold relevant information about assessment products and services that might affect an appropriate selection decision.

2.3

Base all claims about assessment products and services on valid interpretations of publicly available information.

2.4

Allow qualified users equal opportunity to purchase assessment products and services.

2.5

Establish reasonable fees for assessment products and services.

2.6

Communicate to potential users, in advance of any purchase or use, all applicable fees associated with assessment products and services.

2.7

Strive to ensure that no individuals are denied access to opportunities because of their inability to pay the fees for assessment products and services.

2.8

Establish criteria for the sale of assessment products and services, such as limiting the sale of assessment products and services to those individuals who are qualified for recommended uses and from whom proper uses and interpretations are anticipated.

2.9

Inform potential users of known inappropriate uses of assessment products and services and provide recommendations about how to avoid such misuses.

2.10

Maintain a current understanding about assessment products and services and their appropriate uses in education.

2.11

Release information implying endorsement by users of assessment products and services only with the users' permission.

2.12

Avoid making claims that assessment products and services have been endorsed by another organization unless an official endorsement has been obtained.

2.13

Avoid marketing test preparation products and services that may cause individuals to receive scores that misrepresent their actual levels of attainment.

Section 3:
Responsibilities of Those Who Select Assessment Products and Services

Those who select assessment products and services for use in educational settings, or help others do so, have important professional responsibilities to make sure that the assessments are appropriate for their intended use. Persons who select assessment products and services have a professional responsibility to:

3.1

Conduct a thorough review and evaluation of available assessment strategies and instruments that might be valid for the intended uses.

3.2

Recommend and/or select assessments based on publicly available documented evidence of their technical quality and utility rather than on unsubstantiated claims or statements.

3.3

Disclose any associations or affiliations that they have with the authors, test publishers, or others involved with the assessments under consideration for purchase and refrain from participation if such associations might affect the objectivity of the selection process.

3.4

Inform decision makers and prospective users of the appropriateness of the assessment for the intended uses, likely consequences of use, protection of examinee rights, relative costs, materials and services needed to conduct or use the assessment, and known limitations of the assessment, including potential misuses and misinterpretations of assessment information.

3.5

Recommend against the use of any prospective assessment that is likely to be administered, scored, and used in an invalid manner for members of various groups in our society for reasons of race, ethnicity, gender, age, disability, language background, socioeconomic status, religion, or national origin.

3.6

Comply with all security precautions that may accompany assessments being reviewed.

3.7

Immediately disclose any attempts by others to exert undue influence on the assessment selection process.

3.8

Avoid recommending, purchasing, or using test preparation products and services that may cause individuals to receive scores that misrepresent their actual levels of attainment.

       

Section 4:
Responsibilities of Those Who Administer Assessments

Those who prepare individuals to take assessments and those who are directly or indirectly involved in the administration of assessments as part of the educational process, including teachers, administrators, and assessment personnel, have an important role in making sure that the assessments are administered in a fair and accurate manner. Persons who prepare others for, and those who administer, assessments have a professional responsibility to:

4.1

Inform the examinees about the assessment prior to its administration, including its purposes, uses, and consequences; how the assessment information will be judged or scored; how the results will be kept on file; who will have access to the results; how the results will be distributed; and examinees' rights before, during, and after the assessment.

4.2

Administer only those assessments for which they are qualified by education, training, licensure, or certification.

4.3

Take appropriate security precautions before, during, and after the administration of the assessment.

4.4

Understand the procedures needed to administer the assessment prior to administration.

4.5

Administer standardized assessments according to prescribed procedures and conditions and notify appropriate persons if any nonstandard or delimiting conditions occur.

4.6

Not exclude any eligible student from the assessment.

4.7

Avoid any conditions in the conduct of the assessment that might invalidate the results.

4.8

Provide for and document all reasonable and allowable accommodations for the administration of the assessment to persons with disabilities or special needs.

4.9

Provide reasonable opportunities for individuals to ask questions about the assessment procedures or directions prior to and at prescribed times during the administration of the assessment.

4.10

Protect the rights to privacy and due process of those who are assessed.

4.11

Avoid actions or conditions that would permit or encourage individuals or groups to receive scores that misrepresent their actual levels of attainment.

Section 5:
Responsibilities of Those Who Score Assessments

The scoring of educational assessments should be conducted properly and efficiently so that the results are reported accurately and in a timely manner. Persons who score and prepare reports of assessments have a professional responsibility to:

5.1

Provide complete and accurate information to users about how the assessment is scored, such as the reporting schedule, scoring process to be used, rationale for the scoring approach, technical characteristics, quality control procedures, reporting formats, and the fees, if any, for these services.

5.2

Ensure the accuracy of the assessment results by conducting reasonable quality control procedures before, during, and after scoring.

5.3

Minimize the effect on scoring of factors irrelevant to the purposes of the assessment.

5.4

Inform users promptly of any deviation in the planned scoring and reporting service or schedule and negotiate a solution with users.

5.5

Provide corrected score results to the examinee or the client as quickly as practicable should errors be found that may affect the inferences made on the basis of the scores.

5.6

Protect the confidentiality of information that identifies individuals as prescribed by state and federal laws.

5.7

Release summary results of the assessment only to those persons entitled to such information by state or federal law or those who are designated by the party contracting for the scoring services.

5.8

Establish, where feasible, a fair and reasonable process for appeal and rescoring the assessment.

Section 6:
Responsibilities of Those Who Interpret, Use, and Communicate Assessment Results

The interpretation, use, and communication of assessment results should promote valid inferences and minimize invalid ones. Persons who interpret, use, and communicate assessment results have a professional responsibility to:

6.1

Conduct these activities in an informed, objective, and fair manner within the context of the assessment's limitations and with an understanding of the potential consequences of use.

6.2

Provide to those who receive assessment results information about the assessment, its purposes, its limitations, and its uses necessary for the proper interpretation of the results.

6.3

Provide to those who receive score reports an understandable written description of all reported scores, including proper interpretations and likely misinterpretations.

6.4

Communicate to appropriate audiences the results of the assessment in an understand able and timely manner, including proper interpretations and likely misinterpretations.

6.5

Evaluate and communicate the adequacy and appropriateness of any norms or standards used in the interpretation of assessment results.

6.6

Inform parties involved in the assessment process how assessment results may affect them.

6.7

Use multiple sources and types of relevant information about persons or programs whenever possible in making educational decisions.

6.8

Avoid making, and actively discourage others from making, inaccurate reports, unsubstantiated claims, inappropriate interpretations, or otherwise false and misleading statements about assessment results.

6.9

Disclose to examinees and others whether and how long the results of the assessment will be kept on file, procedures for appeal and rescoring, rights examinees and others have to the assessment information, and how those rights may be exercised.

6.10

Report any apparent misuses of assessment information to those responsible for the assessment process.

6.11

Protect the rights to privacy of individuals and institutions involved in the assessment process.

Section 7:
Responsibilities of Those Who Educate Others About Assessment

The process of educating others about educational assessment, whether as part of higher education, professional development, public policy discussions, or job training, should prepare individuals to understand and engage in sound measurement practice and to become discerning users of tests and test results. Persons who educate or inform others about assessment have a professional responsibility to:

7.1

Remain competent and current in the areas in which they teach and reflect that in their instruction.

7.2

Provide fair and balanced perspectives when teaching about assessment.

7.3

Differentiate clearly between expressions of opinion and substantiated knowledge when educating others about any specific assessment method, product, or service.

7.4

Disclose any financial interests that might be perceived to influence the evaluation of a particular assessment product or service that is the subject of instruction.

7.5

Avoid administering any assessment that is not part of the evaluation of student performance in a course if the administration of that assessment is likely to harm any student.

7.6

Avoid using or reporting the results of any assessment that is not part of the evaluation of student performance in a course if the use or reporting of results is likely to harm any student.

7.7

Protect all secure assessments and materials used in the instructional process.

7.8

Model responsible assessment practice and help those receiving instruction to learn about their professional responsibilities in educational measurement.

7.9

Provide fair and balanced perspectives on assessment issues being discussed by policymakers, parents, and other citizens.

Section 8:
Responsibilities of Those Who Evaluate Educational Programs and Conduct Research on Assessments

Conducting research on or about assessments or educational programs is a key activity in helping to improve the understanding and use of assessments and educational programs. Persons who engage in the evaluation of educational programs or conduct research on assessments have a professional responsibility to:

8.1

Conduct evaluation and research activities in an informed, objective, and fair manner.

8.2

Disclose any associations that they have with authors, test publishers, or others involved with the assessment and refrain from participation if such associations might affect the objectivity of the research or evaluation.

8.3

Preserve the security of all assessments throughout the research process as appropriate.

8.4

Take appropriate steps to minimize potential sources of invalidity in the research and disclose known factors that may bias the results of the study.

8.5

Present the results of research, both intended and unintended, in a fair, complete, and objective manner.

8.6

Attribute completely and appropriately the work and ideas of others.

8.7

Qualify the conclusions of the research within the limitations of the study.

8.8

Use multiple sources of relevant information in conducting evaluation and research activities whenever possible.

8.9

Comply with applicable standards for protecting the rights of participants in an evaluation or research study, including the rights to privacy and informed consent.

ASSESSMENT, MEASUREMENT, EVALUATION & RESEARCH

Bill Huitt, John Hummel, and Dan Kaeck

Science: A way of knowing

There are a variety of ways of knowing whether or not something is true. Science is one of those ways; scientists have established a set of rules and methodology by which truth is verified (Kuhn, 1962). The process of science generally follows a paradigm that defines the rules and describes procedures, instrumentation and methods of interpretation of data (Wilber, 1998). The results of science are formulated into a hierarchy of increasing complexity of knowledge: facts, concepts, principles, theories, and laws. When engaged in the process of science, scientists formulate hypotheses or educated guesses about the relationships between or among different facets of knowledge.
Assessment, measurement, research, and evaluation are part of the processes of science and issues related to each topic often overlap. Assessment refers to the collection of data to describe or better understand an issue, measurement is the process of quantifying assessment data, research refers to the use of data for the purpose of describing, predicting, and controlling as a means toward better understanding the phenomena under consideration, and evaluation refers to the comparison of data to a standard for the purpose of judging worth or quality. Assessment and/or measurement are done with respect to variables (phenomena that can take on more than one value or level). For example, the variable "gender" has the values or levels of male and female and data could be collected relative to this variable. Data on variables are normally collected by one or more of four methods: paper/pencil, systematic observation, participant observation, and clinical. Three types of research studies are normally performed: descriptive, correlational, and experimental.
Collecting data (assessment), quantifying that data (measurement), making judgments (evaluation), and developing understanding about the data (research) always raise issues of reliability and validity. Reliability attempts to answer concerns about the consistency of the information (data) collected, while validity focuses on accuracy or truth. The relationship between reliability and validity can be confusing because measurements (e.g., scores on tests, recorded statements about classroom behavior) can be reliable (consistent) without being valid (accurate or true). However, the reverse is not true: measurements cannot be valid without being reliable.
The same statement applies to findings from research studies. Findings may be reliable (consistent across studies), but not valid (accurate or true statements about relationships among "variables"), but findings may not be valid if they are not reliable. At a minimum, for an instrument to be reliable a consistent set of data must be produced each time it is used; for a research study to be reliable it should produce consistent results each time it is performed.

Classification of Scientific Knowledge

The scientific method is used to generate a database of scientific knowledge. A generally accepted hierarchy of scientific knowledge includes:

1. facts -- an idea or action that can be verified -- Example: names and dates of important activities; population of the United States in the latest census;
2. concepts -- rules that allow for categorization of events, places, people ideas, etc. -- Example: a DESK is a piece of furniture (also a concept) designed with a flat top for writing; a CHAIR is a piece of furniture designed for sitting; a CHAIR with a flat surface attached to it that is designed for writing is also called a DESK;
3. principles -- relationship(s) between/among facts and/or concepts; used to generate if-then statements -- Example: the number of children in the family is related to the average scores on nationally standardized achievement tests for those children;
4. laws -- firmly established, thoroughly tested, principle or if-then statement -- Example: a fixed interval schedule for delivering reinforcement produces a scalloping effect on behavior.

Two other important terms relate to how scientists think about and organize this knowledge:

1. hypotheses -- educated guess about what will be found in a scientific study, especially in terms of correlational relationships (if-then statements of principles) and causal relationships (if-then statements of laws) -- Example: for lower-division, undergraduate students study habits is a better predictor of success in a college course than is a measure of intelligence or reading comprehension;
2. theories -- set of facts, concepts, and principles that organize multiple findings and allow for description and explanation -- Example: Piaget's theory of cognitive development, Erikson's theory of socioemotional development, and Skinner's theory of operant conditioning.

The human mind does not think or reason in terms of discrete elements or "facts." Rather the mind seeks patterns among discrete elements and processes information in terms of concepts or the rules for categorizing facts. When people build relationships among facts and concepts (i.e., develop principles), they are able to remember, understand, and access an astonishing amount of information. People are also able to make predictions from present to future circumstances based on these understandings. However, it is when theories are developed (add organization and explanations to facts, concepts, and principles) based on laws (empirically validated causal principles) that scientists accomplish the highest goal of science--to control the variables they are studying.
References:

• Kuhn, T. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.
• Payla, W. (2000). Definition of science. Research methods lecture notes (ver 2.2). Jacksonville, AL: Jacksonville State University. Retrieved May 2001, from http://www.jsu.edu/depart/psychology/sebac/fac-sch/rm/pdfs/Ch2v4.pdf
• Wilber, K. (1998). The marriage of sense and soul: Integrating science and religion. New York: Random House.

Measurement, Assessment, and Evaluation in Education Dr. Bob Kizlik

Throughout my years of teaching undergraduate courses, and to some extent, graduate courses, I was continuously reminded each semester that many of my students who had taken the requisite course in "educational tests and measurements" or a course with a similar title as part of their professional preparation, often had confusing ideas about fundamental differences in terms such as measurement, assessment and evaluation as they are used in education. When I asked the question, "what is the difference between assessment and evaluation," I usually got a lot of blank stares. Yet, it seems that understanding the differences between measurement, assessment, and evaluation is fundamental to the knowledge base of professional teachers and effective teaching. Such understanding is also a core component of the curricula implemented in universities and colleges that are involved in the education of future teachers.

In many places on the ADPRIMA website the phrase, "Anything not understood in more than one way is not understood at all" appears after some explanation or body of information. That phrase is, in my opinion, a fundamental idea of what should be a cornerstone of all teacher education. Students often struggle with describing or explaining what it means to "understand" something that they say they understand. I believe in courses in educational tests and measurements, that "understanding" has often been inferred from responses on multiple-choice tests or solving statistical problems. A semester later, when questioned about very fundamental ideas in statistics, measurement, assessment and evaluation, the students I had seemingly forgot most, if not all of what they "learned."

Measurement, assessment, and evaluation mean very different things, and yet most of my students were unable to adequately explain the differences. So, in keeping with the ADPRIMA approach to explaining things in as straightforward and meaningful a way as possible, here are what I think are useful descriptions of these three fundamental terms. These are personal opinions, but they have worked for me for many years. They have operational utility, and therefore may also be useful for your purposes.

Measurement refers to the process by which the attributes or dimensions of some physical object are determined. One exception seems to be in the use of the word measure in determining the IQ of a person. The phrase, "this test measures IQ" is commonly used. Measuring such things as attitudes or preferences also applies. However, when we measure, we generally use some standard instrument to determine how big, tall, heavy, voluminous, hot, cold, fast, or straight something actually is. Standard instruments refer to instruments such as rulers, scales, thermometers, pressure gauges, etc. We measure to obtain information about what is. Such information may or may not be useful, depending on the accuracy of the instruments we use, and our skill at using them. There are few such instruments in the social sciences that approach the validity and reliability of say a 12" ruler. We measure how big a classroom is in terms of square feet, we measure the temperature of the room by using a thermometer, and we use Ohm meters to determine the voltage, amperage, and resistance in a circuit. In all of these examples, we are not assessing anything; we are simply collecting information relative to some established rule or standard. Assessment is therefore quite different from measurement, and has uses that suggest very different purposes. When used in a learning objective, the definition provided on the ADPRIMA for the behavioral verb measure is: To apply a standard scale or measuring device to an object, series of objects, events, or conditions, according to practices accepted by those who are skilled in the use of the device or scale.

Assessment is a process by which information is obtained relative to some known objective or goal. Assessment is a broad term that includes testing. A test is a special form of assessment. Tests are assessments made under contrived circumstances especially so that they may be administered. In other words, all tests are assessments, but not all assessments are tests. We test at the end of a lesson or unit. We assess progress at the end of a school year through testing, and we assess verbal and quantitative skills through such instruments as the SAT and GRE. Whether implicit or explicit, assessment is most usefully connected to some goal or objective for which the assessment is designed. A test or assessment yields information relative to an objective or goal. In that sense, we test or assess to determine whether or not an objective or goal has been obtained. Assessment of skill attainment is rather straightforward. Either the skill exists at some acceptable level or it doesn’t. Skills are readily demonstrable. Assessment of understanding is much more difficult and complex. Skills can be practiced; understandings cannot. We can assess a person’s knowledge in a variety of ways, but there is always a leap, an inference that we make about what a person does in relation to what it signifies about what he knows. In the section on this site on behavioral verbs, to assess means To stipulate the conditions by which the behavior specified in an objective may be ascertained. Such stipulations are usually in the form of written descriptions.

Evaluation is perhaps the most complex and least understood of the terms. Inherent in the idea of evaluation is "value." When we evaluate, what we are doing is engaging in some process that is designed to provide information that will help us make a judgment about a given situation. Generally, any evaluation process requires information about the situation in question. A situation is an umbrella term that takes into account such ideas as objectives, goals, standards, procedures, and so on. When we evaluate, we are saying that the process will yield information regarding the worthiness, appropriateness, goodness, validity, legality, etc., of something for which a reliable measurement or assessment has been made. For example, I often ask my students if they wanted to determine the temperature of the classroom they would need to get a thermometer and take several readings at different spots, and perhaps average the readings. That is simple measuring. The average temperature tells us nothing about whether or not it is appropriate for learning. In order to do that, students would have to be polled in some reliable and valid way. That polling process is what evaluation is all about. A classroom average temperature of 75 degrees is simply information. It is the context of the temperature for a particular purpose that provides the criteria for evaluation. A temperature of 75 degrees may not be very good for some students, while for others, it is ideal for learning. We evaluate every day. Teachers, in particular, are constantly evaluating students, and such evaluations are usually done in the context of comparisons between what was intended (learning, progress, behavior) and what was obtained. When used in a learning objective, the definition provided on the ADPRIMA site for the behavioral verb evaluate is: To classify objects, situations, people, conditions, etc., according to defined criteria of quality. Indication of quality must be given in the defined criteria of each class category. Evaluation differs from general classification only in this respect.

To sum up, we measure distance, we assess learning, and we evaluate results in terms of some set of criteria. These three terms are certainly connected, but it is useful to think of them as separate but connected ideas and processes.