ASSESSMENT, MEASUREMENT, EVALUATION & RESEARCH

Bill Huitt, John Hummel, and Dan Kaeck

Science: A way of knowing

There are a variety of ways of knowing whether or not something is true. Science is one of those ways; scientists have established a set of rules and methodology by which truth is verified (Kuhn, 1962). The process of science generally follows a paradigm that defines the rules and describes procedures, instrumentation and methods of interpretation of data (Wilber, 1998). The results of science are formulated into a hierarchy of increasing complexity of knowledge: facts, concepts, principles, theories, and laws. When engaged in the process of science, scientists formulate hypotheses or educated guesses about the relationships between or among different facets of knowledge.
Assessment, measurement, research, and evaluation are part of the processes of science and issues related to each topic often overlap. Assessment refers to the collection of data to describe or better understand an issue, measurement is the process of quantifying assessment data, research refers to the use of data for the purpose of describing, predicting, and controlling as a means toward better understanding the phenomena under consideration, and evaluation refers to the comparison of data to a standard for the purpose of judging worth or quality. Assessment and/or measurement are done with respect to variables (phenomena that can take on more than one value or level). For example, the variable "gender" has the values or levels of male and female and data could be collected relative to this variable. Data on variables are normally collected by one or more of four methods: paper/pencil, systematic observation, participant observation, and clinical. Three types of research studies are normally performed: descriptive, correlational, and experimental.
Collecting data (assessment), quantifying that data (measurement), making judgments (evaluation), and developing understanding about the data (research) always raise issues of reliability and validity. Reliability attempts to answer concerns about the consistency of the information (data) collected, while validity focuses on accuracy or truth. The relationship between reliability and validity can be confusing because measurements (e.g., scores on tests, recorded statements about classroom behavior) can be reliable (consistent) without being valid (accurate or true). However, the reverse is not true: measurements cannot be valid without being reliable.
The same statement applies to findings from research studies. Findings may be reliable (consistent across studies), but not valid (accurate or true statements about relationships among "variables"), but findings may not be valid if they are not reliable. At a minimum, for an instrument to be reliable a consistent set of data must be produced each time it is used; for a research study to be reliable it should produce consistent results each time it is performed.

Classification of Scientific Knowledge

The scientific method is used to generate a database of scientific knowledge. A generally accepted hierarchy of scientific knowledge includes:

1. facts -- an idea or action that can be verified -- Example: names and dates of important activities; population of the United States in the latest census;
2. concepts -- rules that allow for categorization of events, places, people ideas, etc. -- Example: a DESK is a piece of furniture (also a concept) designed with a flat top for writing; a CHAIR is a piece of furniture designed for sitting; a CHAIR with a flat surface attached to it that is designed for writing is also called a DESK;
3. principles -- relationship(s) between/among facts and/or concepts; used to generate if-then statements -- Example: the number of children in the family is related to the average scores on nationally standardized achievement tests for those children;
4. laws -- firmly established, thoroughly tested, principle or if-then statement -- Example: a fixed interval schedule for delivering reinforcement produces a scalloping effect on behavior.

Two other important terms relate to how scientists think about and organize this knowledge:

1. hypotheses -- educated guess about what will be found in a scientific study, especially in terms of correlational relationships (if-then statements of principles) and causal relationships (if-then statements of laws) -- Example: for lower-division, undergraduate students study habits is a better predictor of success in a college course than is a measure of intelligence or reading comprehension;
2. theories -- set of facts, concepts, and principles that organize multiple findings and allow for description and explanation -- Example: Piaget's theory of cognitive development, Erikson's theory of socioemotional development, and Skinner's theory of operant conditioning.

The human mind does not think or reason in terms of discrete elements or "facts." Rather the mind seeks patterns among discrete elements and processes information in terms of concepts or the rules for categorizing facts. When people build relationships among facts and concepts (i.e., develop principles), they are able to remember, understand, and access an astonishing amount of information. People are also able to make predictions from present to future circumstances based on these understandings. However, it is when theories are developed (add organization and explanations to facts, concepts, and principles) based on laws (empirically validated causal principles) that scientists accomplish the highest goal of science--to control the variables they are studying.
References:

• Kuhn, T. (1962). The structure of scientific revolutions. Chicago: University of Chicago Press.
• Payla, W. (2000). Definition of science. Research methods lecture notes (ver 2.2). Jacksonville, AL: Jacksonville State University. Retrieved May 2001, from http://www.jsu.edu/depart/psychology/sebac/fac-sch/rm/pdfs/Ch2v4.pdf
• Wilber, K. (1998). The marriage of sense and soul: Integrating science and religion. New York: Random House.