Theodore W. Frick
School of Education
Indiana University Bloomington
Last revised April 20, 1997
Note: This document elaborates the article,
Artificial Tutoring Systems:
What Computers Can and Can't Know
Journal of Educational Computing Research, Vol. 16(2),
107-124, 1997.
https://doi.org/10.2190/4CWM-6JF2-T2DN-QG8L
Introduction
Overview of Maccia's Epistemology
1. Qualitative Intelligence: Knowing That-One
1.1. Recognition
1.2. Acquaintance
1.3. Appreciation
Implications of Qualitative Knowing for Intelligent Tutoring
Systems
2. Quantitative Intelligence: Knowing That-Any
2.1. Instantiation
2.2. Theoretical knowing
2.3. Criterial knowing
Implications of Quantitative Knowing for Intelligent Tutoring
Systems
3. Praxic Intelligence: Knowing How-to-Do
3.1. Protocolic procedures
3.2. Protocolic performances
3.3. Conventional procedures
3.4. Conventional performances
A word of caution
Implications of Praxic Knowing for Intelligent Tutoring
Systems
4. Inventive Intelligence: Knowing What-to-Do
4.1. Innovative know-how
4.2. Creative know-how
Implications of Inventive Knowing for Intelligent Tutoring
Systems Conclusion Notes Note 1
Note 2
Note 3
Note 4
References
George Maccia created his epistemology of educational objectives as an alternative to the widely-used taxonomy of cognitive objectives that Benjamin Bloom and his colleagues developed.
Maccia's epistemology includes three basic kinds of knowing:
In this document, I discuss in detail and provide examples of Maccia's epistemology since it is not widely known at this time. Moreover, I illustrate its value in helping us understand the limitations of artificially intelligent tutoring systems. By building on the work of scholars such as Dewey, Peirce, Scheffler, Bateson, Steiner and Maccia, I attempt to demonstrate from an epistemological perspective why these problems exist in AI, and why they are not likely to be solved with present-day computer systems.
Given what computers can and cannot know at this time, then teachers and administrators can plan intelligently for how computers can be used in education. A further benefit of this epistemological perspective is that it may help educators to think differently about what we expect students to learn in school, and it does provide useful criteria for assessing such learning outcomes.
First, we should clarify some basic terms: believing, knowing, cognition, learning, knowledge, and intelligence. One might ask why. Steiner (1988; 1981) discusses at length the importance of defining terms if theory construction and testing is to be a disciplined endeavor. In physics and chemistry, for example, terms such as atom, molecule, hydrogen, electron, mass, energy, momentum, velocity, acceleration, etc. have specific meanings to researchers in these fields. Steiner argues that knowledge of education, which she calls 'educology,' will not progress significantly until educational researchers do come to some consensus on the meanings of terms used in educational theorizing. It is apparent that this is generally not the case, and therefore not surprising that those who do educational inquiry often get little respect from researchers in the more established disciplines such as physics, biology, chemistry and medicine.
In this article, I am using certain terms in narrow, but specific ways that are generally consistent with Occidental philosophers whose ideas have remained vital for some time, including Socrates, Plato, Aristotle, Dewey, Peirce, Steiner and Maccia. While it is true that we can negotiate and construct whatever meanings we want for our terminology (cf., Duffy & Jonassen, 1992), there is obvious value for holding meanings of specific terms constant over time. Imagine, for example, the chaos that would prevail in medical science if epidemiologists were inconsistent in usage of terms for diseases.
In the paragraphs that follow, I attempt to describe relationships among terms which are relevant to epistemology.
To believe something is not necessarily to know it. For example, I may believe that it is snowing outside, but not know it. To justify my belief -- i.e., to know it -- I need to go see for myself, or be informed by witnesses whom I trust do know what they are talking about. Cognition includes both knowing and believing.
When we 'know' something, we hold a belief that has been verified in a disciplined way. When we record what we know (e.g., in writing, by making a video), then it becomes 'knowledge'. Knowledge is recorded knowing (Steiner, 1988).
Disciplined inquiry is the rational way to settle doubt and so to fixate belief (C.S. Peirce, 1958). Disciplined inquiry contrasts with the methods of tenacity, authority and agreeableness to reason.
In the method of tenacity, one continues to hold on to a belief, despite evidence to the contrary. The method of tenacity might be construed as an operational definition for ignorance. For example, to continue to believe that the earth is flat when shown a satellite photograph of our planet, would be to use the method of tenacity.
When one uses the method of authority to fixate belief, it is on the basis of what someone or some group in a position of power or influence believes. For example, to believe that the earth is the center of the universe and that the sun revolves around it because that was the official teaching in the Catholic Church for many years, is to use the method of authority. Galileo was branded a heretic, because he held a different belief that was supported by evidence from astronomical observation.
The method of agreeableness to reason is to believe in an untested theory or hypothesis. For example, it sounds reasonable that humans with larger brains would be expected to be more intelligent than those with smaller brains. It stands to reason that if there is more cortical matter, then one could remember more and hold more thoughts. Unfortunately, while this may sound reasonable, such a belief is not supported by empirical evidence.
Disciplined inquiry is the method of science, the method of knowing. Disciplined inquiry is the alternation of theory development, theory testing (verification), and making a record of those results for others to inspect, evaluate, and to learn from (cf. Peirce, 1977; Steiner, 1988).
When we hold a belief about something that has not been verified in a disciplined manner, then such a belief is an opinion or a conjecture (cf., Socrates). Even if the belief is well-thought out and called a theory, without adequate evidence to support it, such a belief remains a theory.
To learn something is to come to know it or to believe it (cf., Aristotle and Dewey). Learning is a process. The 'something' is the object of knowing or believing. Cognition is psychical construction -- a relationship that is formed between one's mind and the object of knowing (cf., Bateson, Bruner, Dewey). This does not imply that knowledge is transmitted, like pouring water into a bucket. Dewey discounted this notion long ago. As we reflect about objects of experience -- i.e., think -- we somehow integrate these thoughts into our private mental schemes of things. This integration, what we know and what we believe, is an outcome of the learning process (Note 3).
In contrast to knowing, knowledge is a set of public 'signs' shared among conscious entities which represent states of affairs (cf. Bateson, 1979; Bruner, 1990; Peirce, 1977; Steiner, 1978). Knowledge is neither exclusively subjective nor solely objective, but instead intersubjective. 'Signs' are the "stuff" of communication. Signs can include spoken and written language (linguistic symbols), icons, gestures, pictorial representations, non-linguistic sound patterns, smells, touch, facial expressions, expression of emotion, and demonstration by enactment. Signs take on meaning for humans through use during situated action (Bruner, 1990; Frick & Reigeluth, 1992).
Knowledge must be distinguished from the objects and their interrelationships which are represented. There are objects in the empirical realm and in the abstract realm.
Empirical objects of knowledge. Such objects can be perceived through our senses or through instruments which amplify our senses. Sticks and stones and broken bones are examples such objects. We can contemporaneously perceive such objects through sight, hearing, touch, taste and smell.
Abstract objects of knowledge. Such objects cannot be perceived through our senses but can be apprehended mentally. The concept of 'zero' is an abstract object; likewise, the concept of 'justice'. Ideas exist in people's minds. They are just as real as sticks and stones. However, we cannot apprehend others' thoughts directly.
We cannot ordinarily tell what humans know or believe just by observing them, somehow peering into their minds. Humans can disclose what they know or believe through use of 'signs.' The signs are public -- i.e., intersubjective -- shared between persons. Knowing and believing are private properties of an individual conscious entity.
Finally, intelligence is taken as ability to learn.
Maccia makes distinctions among four classes of knowing: Qualitative knowing pertains to that-one, to what is unique. Quantitative knowing pertains to that in general, that which is not bound in space and time. Praxic knowing pertains to procedures and being able to perform them smoothly to accomplish some end. Inventive knowing pertains to innovation and creation.
These four classes are further subdivided, resulting in twelve kinds of cognition. Qualitative intelligence consists of recognition, acquaintance and appreciation. Quantitative intelligence is characterized by instantial, theoretical and criterial knowing. Praxic intelligence is comprised of protocolic-procedural, protocolic-performative, conventional- procedural and conventional-performative know-how. Finally, innovation and creation are the two kinds of inventive intelligence.
G. S. Maccia's epistemology is built on the work of other important philosophers, including Socrates, Peirce, Dewey, Scheffler and Steiner. He had not formally published his work, other than those works cited, when he retired under duress from Indiana University in December, 1989. Educational philosophy was no longer a separate department in the School of Education. Imagine how physicists or biologists would feel if their departments lost their identities, after spending a lifetime of productive work in their respective disciplines. Recall that Socrates was forced to drink the hemlock because his views were not politically correct at the time. Moreover, it is not safe to infer that because a large number of people hold a common belief, that the belief is right. Historically, most people believed that the earth was flat and the center of the universe for millennia. Copernicus, Kepler and Galileo saw things differently about 500 years ago.
As you read about Maccia's epistemology, test it on yourself. Think of what you know in relation to these twelve kinds of cognition. After all, the test is not what others think -- since you cannot see into their minds -- but whether this epistemology is consistent and coherent within your own mental experience. I also do so in this article, in order to provide supplementary examples.
Qualitative intelligence is ability to index 'none-other,' that is, to discern the specific features that make some entity unique, what sets it apart from all else. For example, to know in one's residence where each piece of furniture is located, where dishes are stored in the kitchen, where items of clothing can be found in various closets, etc., is 'knowing that-one.' Many people have such accurate kinesthetic and mental images of where things are that they can walk around their residences in the dark.
Maccia further categorizes qualitative intelligence into three types: recognition, acquaintance and appreciation.
Qualitative recognition is discernment of that-one from all else. For example, we can recognize our Earth from a wide-angle photograph taken from a satellite, distinguishing it easily from other bodies such as our moon, Jupiter, Venus, etc. A further example is that even not having seen him for some time, I had no difficulty recognizing my brother in a crowded airport terminal several years ago. During the approach to landing at National Airport in Washington, D.C., I recognized the Capitol Building and the Washington Monument. Recognition is not limited to vision. For example, I immediately recognize my wife's voice on the telephone when she calls; she seldom needs to identify herself.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) has learned to recognize Q, the object of knowing:
S recognizes Q if and only if
1. S believes that Q.
2. S is completely justified in believing that
Q.
3. No other statement or belief defeats S's
belief that Q.
4. S selects Q from not Q and not Q from Q.
5. Q is a state of affairs.
6. T knows that the above conditions hold in
order that S
recognize Q.
One of Maccia's conditions of recognition is that no other statement or belief defeats S's belief in some unique state of affairs. A good example of how to test this is a technique used by attorneys when questioning witnesses during a legal trial. In attempting to refute positive identification, an attorney will try to raise doubts in the mind of the witness of whether he or she is certain the defendant was the one at the scene of the crime. Or police detectives may put a suspect in several different line-ups to test whether a witness can make a positive identification of the suspect consistently.
Qualitative acquaintance is identification of those particular relationships which determine the uniqueness of some entity or state of affairs. For example, if one is acquainted with the Washington Monument she or he could describe its unique qualities, such as its being an obelisk several hundred feet tall, dedicated as a memorial to the first U.S. president, which is located in the center of a cross-shaped mall, respectively ended by the Lincoln Memorial, the Jefferson Memorial, the White House and the Capitol Building in Washington, DC, on the North American Continent, planet Earth. These specific relationships distinguish the Washington Monument from all else.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) has become acquainted with Q, the object of knowing:
S is acquainted with Q if and only if
1. S recognizes Q.
2. S selects elements [qi ... qj] determinate of
Q; and
relations [ri ... rj] determinate of Q.
3. Q is a state of affairs.
4. T knows that the above conditions hold in
order that S be
acquainted with Q.
Qualitative acquaintance requires more than recognition, which is necessary for acquaintance. Acquaintance also requires that a student select elements and relations which are determinate of a state of affairs Q i.e., what makes it unique. For example, if a student were acquainted with George Washington's life history, she or he would know that he lived at Mount Vernon in Virginia in the mid- to late 18th century, that he was the commander of the American army which defeated the British during the revolution for American independence, and that he was the first president of the United States. He is honored in many ways, probably most significantly by his engraved image on the U.S. one-dollar bill and by the Washington Monument in the nation's capital.
If one were acquainted with Beethoven's Symphony No. 9 in D Minor, Opus 125, she or he would know that it is comprised of four movements, the last of which requires a large choral group to accompany the orchestra and four vocal soloists, and so forth. If the student were presented with musical excerpts she or he could say which section of which movement the excerpt was taken from. If acquainted with Beethoven's life, a student would know that he was deaf when composing his ninth and last symphony.
The above two paragraphs contain examples of selected elements and relations regarding the life of George Washington and Beethoven's Ninth Symphony, respectively. If a student's description is sufficient, we would infer that he or she is acquainted with that-one.
Qualitative appreciation is perspicacious evaluation of those particular elements and their relationships which are appropriate of a unique entity or state of affairs. Appreciation requires acquaintance, but is more than just knowing specific relationships which determine uniqueness. Appreciation also requires sensitivity. For example, a film or music critic often evidences appreciation when writing an evaluation of a particular movie or concert for a news column.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) appreciates Q, the object of knowing:
S appreciates that Q if and only if
1. S is acquainted with Q.
2. S selects elements [qi ... qj ] appropriate of
Q; and
relations [ri ... rj] appropriate of Q.
3. Q is a state of affairs.
4. T knows that the above conditions hold in
order that S
appreciate that Q.
What has been said above also applies to assessment of student appreciation. This is because appreciation requires acquaintance, and acquaintance in turn requires recognition. Appreciation goes one step further. A student must select elements and relations which are appropriate of a unique state of affairs (i.e., fitting, proper, relevant). Take, for example, Olympic diving. A judge appreciates a particular dive by observing and looking for certain features such as height of take- off; form during tucked positions, rotations and twists; smoothness and verticality of entry into the water; lack of splash; difficulty of the dive; and the diver's ability to carry out the intent of the dive (the kind of dive she or he was trying to do). Such judgment or appreciation can be learned.
Film critics evidence appreciation of particular movies. A well- known pair of critics, Siskel and Ebert, regularly evidence their appreciation by verbally discussing on TV the qualities of recently released films. They do not always agree in their ultimate conclusions (i.e., thumbs up or down), but they tend to use similar criteria in their evaluations such as the quality of the story line, pacing of the action, quality of acting, and the quality of the screenplay itself (visual effects, camera angles, transitions,, special effects, sound track). What is most important are their descriptions of their judgments of the qualities of a particular film, i.e., why the story line dragged, what about the acting that made it superb, etc.
To assess student appreciation it appears necessary to have the student directly experience the state of affairs Q and then to prepare and deliver an oral or written report. This way a teacher can observe what the student has selected with respect to appreciation of that unique state of affairs. The teacher can judge the appropriateness, relevance, and fittingness of what the student has selected to present.
If an intelligent tutoring system contained qualitative intelligence, then one of them might tease me as I approached it, "Oh, it's you again, Ted. Don't try to tell me what I don't know." Humans and some animals possess varying degrees of qualitative intelligence; whether computer systems can is another question -- it is more than just pattern recognition. Let me give a personal example. When I arrive home and drive my car up our long driveway, my cats which live outdoors come and greet me. They do not do this when strangers come up the driveway. This is evidence of the cats' qualitative recognition of me and my car.
Thus far, none of the many computer systems on which I have worked have evidenced this kind of capability, let alone acquaintance and appreciation. In fact I must input information (e.g., when I log in) in order identify myself. But that is not recognition on the part of the computer system. Anybody can assume my "identity" by entering my account name and password. It would be harder to impersonate me if a retinal scan or an image of my thumbprint were provided as input to a computer. But notice that it is we humans who have done the selection of unique features to be provided as input, rather than active perception by a computer system.
In order to teach a student to know qualitatively, an artificially intelligent tutoring system would itself need to know that-one. Otherwise, how could it tell if student learning is successful? For example, if a computer system is not itself capable of recognizing the Washington Monument, then how would it be able to tell if a student is able to do so? This does not mean that computers cannot be used to guide qualitative learning. A human tutor can certainly use a computer system as a medium to help bring students to know qualitatively.
Indeed, with today's multimedia capabilities, a teacher or instructional developer could videotape the Washington Monument, to continue the example, and present information much like a tour guide. The computer could be instructed by the teacher to play various video segments when a student interacts with the computer system. The computer could also be programmed to test the student's knowledge of the Washington Monument by presenting pictures of various monuments, tombstones and buildings. The computer system could be instructed further to determine if the student can correctly point out which pictures are those of the Washington Monument and which are not. All this is currently possible with today's technologies.
However, the computer system itself would still not be able to recognize the Washington Monument in the above example. If the human teacher were mistaken, and instead videotaped the Lincoln Memorial but incorrectly called it the Washington Monument and likewise instructed the computer to test for this misinformation, the computer system would not know that the information was incorrect and would reliably follow the tutor's instructions. "Garbage in, garbage out," as the saying goes. If an unfortunate student were to encounter this bogus instruction, he or she might learn to recognize the Lincoln Memorial, but would falsely believe that it is the Washington Monument. The computer is blind with respect to the content and only knows how to carry out the instructional developer's intention.
Present-day computer systems are fundamentally a medium for conveying representations of what humans know, believe, feel and intend -- a medium which can nonetheless follow directions and logic it is given. In this case, the teacher's belief was mistaken, and we would conclude that the teacher himself or herself did not know that-one, the Washington Monument.
Quantitative intelligence is the ability to 'know that' and is "usually associated with mental acts that employ abstractive inference (i.e., modes of generalization or instantiation)." (Maccia, 1987, p. 215) Such knowing is not bound to particular persons, places, events, things or their interrelationships. For example, to categorize Earth, Mercury, Venus, Mars, etc. as instances of the concept 'planet' is quantitative instantiation. Or to know the relationship between matter, energy and light, as expressed in Einstein's famous equation, E = mc2, is also quantitative. To know the value 'freedom of speech' is likewise an example of quantitative (or generic) intelligence.
In logical discourse, 'quantification' implies extension relative to class, not just in the narrow sense of counting or measuring. In other words, if x is true of b1, b2, and b3, and these b's are representative members of some larger class B, then x is likely to be true of all members of B. Einstein's equation (x), is not only true of our sun (b1), but of all matter and energy in the universe (B).
Maccia further distinguishes three kinds of quantitative intelligence: instantiation, theoretical knowing and criterial knowing.
Quantitative instantiation is classification of some state of affairs according to type or kind -- i.e., grouping according to commonality or similarity. Identifying Venus as an example of a planet is quantitative instantiation. Identifying murder as an instance of injustice is likewise quantitative instantiation.
The kinds of states of affairs are not restricted to concepts, but also can include relationships, patterns and rules. An apple falling to the earth is an instance of Newton's laws of gravity, which characterize relationships among concepts such as mass, force, velocity, acceleration, distance and time. The on-going nuclear reaction on the sun is an instance of the relationship between matter and energy, as expressed by Einstein's famous equation above.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) can instantiate Q, the object of knowing:
S instantiates Q if and only if
1. S believes that Q.
2. S identifies Q as an instance of a kind.
3. S correctly believes Q.
4. Q is a state of affairs.
5. T knows that the above conditions hold in
order that S
identify Q.
One of Maccia's conditions in instantial knowing is that S correctly believes that Q. In instantial knowing a student identifies Q, a state of affairs, as an instance of a kind; and she or he does so correctly. A student may believe that a whale is a fish. Such a belief is incorrect because a whale is an instance of mammal.
On the other hand, to identify a dog or cat as an instance of an animal, or a tree or grass as an instance of a plant, would be evidence of quantitative instantial knowing since such beliefs are correct. If the goal of instruction were for a student to learn the concept of 'right triangle,' he or she could be presented with a group of geometric figures and point out which were right triangles. Or if the goal were to learn to discriminate vertebrates from invertebrates, the student could be presented with pictures of animals and be required to label them as vertebrates or invertebrates.
In instantial knowing, the generalities could be concepts, rules, laws, patterns, principles or relationships. It is more than concept learning as it is usually conceived (cf. Leshin, Pollock & Reigeluth, 1992; Merrill, 1983; Gagne, 1985). The basic task is to classify instances into their appropriate categories, which means to sort or label them correctly.
Assessment of instantial knowing is similar to that of recognition. The difference is the distinction between the unique versus a generalization. To select pictures of George Washington from those of others is to recognize a unique individual (qualitative recognition). If instead the student is given a box of marbles and asked to sort them into piles of basic colors such as red, blue, yellow, green, magenta, white, black and cyan, then we are asking the student to make generalizations (i.e., quantitative instantiations). This marble is an instance of red. That marble is an instance of green, and so forth.
Quantitative theoretical knowing is understanding of the part-part and part-whole relationships of a theory which is generalizable -- i.e., an interrelated set of propositions about 'what generally is.' An example of theoretical quantification would be to provide arguments and evidence that justifies Newton's theory of gravity, as well as explicate its relevance and fruitfulness. Einstein's 1961 book, Relativity, is another example, par excellence.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) knows the theory of that Q, the object of knowing:
S knows the theory of that Q if and only if
1. S believes that Q.
2. S is in a position to know that Q.
3. S correctly believes that Q.
4. S presents an evidentiary argument that
completely
justifies S's belief that Q.
5. S explicates the relevance and fruitfulness
of the theory
of that Q.
6. Q is a state of affairs.
7. T knows that the above conditions hold in
order that S
knows the theory of that Q.
A further example of theoretical quantification is the content of this article you are now reading. I am attempting to provide evidentiary arguments to justify Q, Maccia's theory of natural intelligence. I am also discussing its relevance and fruitfulness regarding artificially intelligent tutoring systems.
Quantitative criterial knowing is valuation of what is in general worthwhile -- i.e., 'what ought to be.' It concerns the relative importance of values or criteria. Such criteria could be employed in making judgments among entities or states of affairs in general. In other words, criterial quantification is philosophic understanding -- i.e., knowing what is worthwhile.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) knows the criteria of that Q, the object of knowing:
S knows the criteria of that Q if and only if
1. S believes that Q.
2. S is in a position to know that Q.
3. S correctly believes that Q.
4. S presents a justificatory argument to
establish the
credibility of criteria of that Q.
5. S demonstrates the relevancy and fruitfulness
of criteria
of that Q.
6. Q is a state of affairs.
7. T knows that the above conditions hold in
order that S
knows the criteria of that Q.
An example of criterial intelligence would be evidenced by justifying the criteria underlying the Bill of Rights and also to demonstrate their relevance and fruitfulness. A further example is evidenced by attempts of Socrates and his fellow inquirers to define and understand the criterion of 'justice' in The Republic of Plato. Justice is a criterion which pertains not to one person but to all of humankind, and can be used to evaluate and compare various social and governmental systems. A final example would be a woman's 'right to choose' an abortion versus an unborn child's 'right to life.' To be able to argue which criterion is more important, under what conditions, and why is an example of criterial knowing.
If an artificially intelligent tutoring system were to possess generic cognition, then it would cognize that "that's a tree and I'm a computer" -- i.e., it would know concepts. It would also think and reason abstractly. For example, it could understand the phenomenon of gravity assist, whereby space vehicles can pass nearby a planet to increase or decrease their velocity and change direction. It could also understand Maccia's genetic epistemology of intelligent natural systems.
Present-day computer systems can do some kinds of abstract reasoning. The best examples are probably in the area of applied intelligence referred to as 'expert systems.' Expert systems have been developed, for example, to help physicians identify types of bacterial infections, to aid investor decisions on buying and selling stock, for aid in assembling components of computer systems, for making decisions about where to drill for oil, for assisting underwriters in making insurance policies, and for diagnosing causes of equipment failures to help repair persons.
Early expert systems consisted of sets of production rules or frames, often called 'knowledge bases.' The name, 'expert system,' was coined because a so-called knowledge base was typically constructed in early expert systems by interviewing one or more experts in some domain of knowledge. An attempt was made to capture their reasoning processes, when they solve problems in that knowledge domain, in the form of "If..., then..." rules. For example, in MYCIN, a famous early expert system for diagnosing bacterial infections, one of the rules is:
IF 1) the gram stain of the organism is negative, and
2) the morphology of the organism is rod, and
3) the aerobicity of the organism is anaerobic,
THEN there is suggestive evidence (.7) that the identity of the organism is Bacteroides. (Davis, 1984, p. 34)
This particular rule is one of over 400 such rules that comprise the MYCIN knowledge base. A computer program, called an 'inference engine,' uses this rule set as data to help physicians identify unknown bacteria. The program makes categorical inferences by using both the rule set and specific answers to questions it asks a physician about properties of the current situation (e.g., patient symptoms, white blood cell count, and other lab test results).
I believe that the term, 'knowledge base' is a misnomer, because a computer system does not understand the rules that represent the reasoning of a human expert. It does however know how to execute the rules. That is what the inference engine does. The inference engine is, in turn, a computer program, a set of instructions which the computer follows. As discussed above, "garbage in, garbage out," applies to expert systems as well. If the reasoning of the human expert is incorrect, the reasoning of the computer's inference engine will be incorrect as well.
Indeed, an expert system is not an example of quantitative knowing, but instead an example of know-how, which is discussed in the next section. What makes matters confusing is that it is easy to conflate knowing and use of language. When we say someone does not know what he is talking about, what we mean is that he is putting together words in ways that are linguistically acceptable but which do not correspond with facts or the truth of the matter (e.g., like the teacher above who mistook the Lincoln Memorial for the Washington Monument). We should not confuse understanding (theoretical and criterial knowing) with manipulation of symbol systems.
A computer system which possesses quantitative intelligence would also be able to understand moral values -- what is right or good. For example, in several of Isaac Asimov's science fiction novels he emphasizes the three laws of robotics:
1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm.
2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Law. (Asimov, 1964, p. 43)
The robotic computers in Asimov's novels evidenced criterial intelligence, although they were not supposed to be able to evaluate and modify these three criteria through reasoned communal discourse.
In the late Asimov's novel, Robots and Empire (1985), the telepathic robot Giskard concluded that a new law, the Zeroth Law, took precedence over the first three: "Prevention of harm to human beings in groups and to humanity as a whole comes before the prevention of harm to any specific individual." (p. 463) If we humans ourselves were as beneficent and just, exhibiting such criterial intelligence as did Giskard, the world would be a much better place to live.
Present-day computer systems are extremely limited in what they can know generically, although they can certainly be used as media for teaching generalizable knowledge, as discussed above for qualitative intelligence. A human teacher can direct a computer system to present generalities, theories and criteria, and she or he can teach it how to test student acquisition of instantial knowledge. Assessment of theoretical and criterial knowing is another matter, however, because students are required to explicate (i.e., speak and/or write about) the theories and criteria as well as justify them on the basis of evidence, relevance and fruitfulness. Imagine that a computer system were filled with a dictionary of the English or any other language. That is, for each word it can quickly retrieve its definition, or one of its definitions if there are more than one. Now imagine that the student writes a paper to demonstrate his or her understanding, for example, of Darwin's theory of evolution. The computer could take each word in the paper and substitute its definition. Now the paper would have become much longer than what the student originally wrote. The computer next substitutes definitions of all the words in the previously substituted definitions, and the paper expands even further in terms of the total number of words. This exercise could go on indefinitely.
The reason for this problem is that a dictionary is a circular definitional system. Words are defined in terms of each other. That is why it is impossible to learn a new language such as Chinese by attempting to memorize a Chinese dictionary. Dictionaries are only useful if we already know some of the language to begin with; this is how we get out of the circularity problem.
Logicians would say that 'primitive terms' are needed in a definitional system to prevent circularity (cf. Steiner, 1988). 'Primitive terms' are undefined. Other terms can then be defined on the basis of these primitive terms, and further 'defined terms' can use previously defined terms. This would be a way for the computer to get out of the never-ending substitution task described above, because substitution would stop for any given defined word when it eventually reaches a definition consisting of solely primitive terms -- assuming that the definitional system contained all the words in the student's original paper.
The crux of the matter is reached at the primitive terms. Since they are undefined, then how can the computer system know what they mean in order to understand what the student is saying?
When we human beings learned our native language as children, we associated those signs (i.e., words) with things we experienced. 'Dogs,' 'cats,' 'grass' and 'trees' were words which had meaning because we saw them, touched them, smelled them, heard them and probably played with the cats and dogs in the grass and climbed some of those trees. No one needed to define these primitive terms for us, and if they had we might have gotten really confused. They instead used language in these situated life activities. When we later learned the more abstract concepts of 'animals' and 'plants,' those terms were meaningful because we could connect them to specific life experiences which were instances of those concepts. In short, concepts without percepts are meaningless (cf. Dewey, 1916).
Present-day computer systems might appear to possess quantitative knowledge, but we must seriously question whether they can know quantitatively to any large extent. Educators can nonetheless use such systems as media to help students come to know generalizations.
Praxic intelligence is the ability to 'know how,' to use means to achieve ends. For example, we can read printed text, construct superhighways, play tennis, perform open-heart surgery, and know how to do research. Know-how requires cognition of particular circumstances, making judgments based on the conditions, and choosing appropriate courses of action when warranted -- all in order to achieve some desired outcome. In short, know-how is purposeful.
Maccia further characterizes praxic intelligence as: protocolic- procedural, protocolic-performative, conventional-procedural, and conventional-performative.
The distinction between procedural and performative knowing is essentially the difference between iterating a set of steps to achieve some outcome versus actually carrying them out in a smooth manner. For example, listing the steps in baking a cake (the recipe) is evidence of procedural knowing. Actually making the cake is evidence of performative knowing.
The distinction between protocolic versus conventional knowing is whether the procedure/performance is single-pathed or multi-pathed. In a single-pathed doing, there is only one correct way. For example, properly opening a combination lock is a protocolic performance. On the other hand, for conventional know-how there are different ways to reach the same end. To add two numbers, for instance, we can count on fingers, use an abacus or calculator, or do it in our heads.
Examples: If a student knew the procedure for opening a particular combination lock, he or she would be able to list the steps in the proper order e.g., spin the dial several times clockwise, then turn it right to 12, left to 45 and right to 23. Or, if a student knew the procedure for making chocolate chip cookies, she or he could list the ingredients and steps to take.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) knows the protocolic procedure of P, the doing:
S knows the protocol of P if and only if
1. S iterates the constituents and succession of
movements
in executing the protocol.
2. The protocol is the way of performing P.
3. P is a single-pathed doing.
4. T knows that the above conditions hold in
order that S
knows the protocol for doing P.
Knowing the procedure does not guarantee that a student can perform it successfully. To continue with the above example, he or she would need to actually open the combination lock on several different occasions, in order to provide evidence of protocolic- performative know-how. Or he or she would actually need to make and bake a batch of chocolate chip cookies, by following a specific recipe. Playing the Mozart Clarinet Concerto for a live audience would be a further example.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) knows how to do the protocolic performance, P:
S knows how to do the protocol P if and only if
1. S has the capacity for doing P.
2. S has the facility for doing P.
3. S smoothly executes P.
4. P is a single-pathed doing.
5. T knows that the above conditions hold
for doing P.
One condition of performative knowing is that S has the capacity for doing P. For example, an infant does not have the capacity to open a combination lock. He or she most likely lacks the necessary motor coordination and cannot read the numbers on the dial. Or if a student does not know how to measure wholes and fractions of teaspoons, tablespoons, and of cups of ingredients, then she or he would lack the capacity for making chocolate chip cookies.
Another condition is that S has the facility for doing P. The performance must be done with ease, efficiently and proficiently. If it takes the student 6 hours to make a batch of chocolate chip cookies, the performance would not be facile.
The doing must be also be smooth. If the student requires several attempts before successfully opening the lock, because he or she has difficulty lining up the pointer with the numbers on the dial, the performance would not be smooth.
The difference between conventions and protocols is that there is more than one way to do a conventional procedure. A good example is going from one place to another geographically. There are usually several different routes that can be taken. If a student were to describe how to get from the Indianapolis Airport to the Bloomington Campus of Indiana University, she or he could describe a route which was correct. But there is more than one route that can be taken and still succeed in the task.
Another example of conventional-procedural know-how would be to describe steps one would go through to determine the fault in a car that will not start (or some other piece of equipment). There are usually several different ways that this could be done, depending on the test equipment available and the nature of the problem.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) knows the conventional procedure, P:
S knows the convention of P if and only if
1. S iterates the preferred constituents and
succession of
movements in executing P.
2. The convention is a way of performing P.
3. P is a multi-pathed doing.
4. T knows that the above conditions hold in
order that S
knows the convention for doing P.
As above, S must actually be able to perform the task, but there is more than one correct way to do it. For example, a student could drive a car from the Indianapolis Airport to the IU campus, or she or he could ride a bicycle. As another example, if we want to assess whether the student can solve algebraic equations with one unknown, he or she can be presented with such equations. If the student arrives at correct answers most of the time and fairly quickly without a lot of effort, then we would conclude that he or she has mastered this objective. There are usually several different ways this can be done, however, and so this is a conventional performance.
Maccia (1989) indicates that the following conditions must be met in order for a teacher (T) to determine that a student (S) knows how to do the conventional performance, P:
S knows how to do the convention P if and only if
1. S has the capacity for doing P.
2. S has the facility for doing P.
3. S smoothly executes P.
4. P is a multi-pathed doing.
5. T knows that the above conditions hold for
doing P.
Know-how does not necessarily involve a physical or motoric doing. Adding numbers, which one can do in his head, is an example of know-how. The mind-body distinction is irrelevant. The doing is mindful. You must know how to read English in order to understand this text. The turning of pages is incidental. The ability to decode the squiggles on the page is essential to such know-how. The means is reading; the end could be thinking, believing, knowing, or simply enjoyment. That is, reading is one thing; comprehension is another. Reading is a means to cognition. Kinds of cognition (i.e., knowing) are characterized in this article.
Computers are very good at following protocols such as instructions for computation and logical operations. When such instructions are properly sequenced, a computer can do complex tasks such as flying airplanes, multivariate statistical analysis, medical diagnosis, buying and selling of stock, diagnosing faults in equipment, anti-lock braking to stop a vehicle, etc. To date, computer systems have evidenced more ability in the domain of praxic intelligence, compared to any of the others. This should not be surprising, since computation is an example of know-how. It is also not surprising that those artificially intelligent tutoring systems which have been successful usually teach know-how (e.g., solving algebraic equations, diagnosing equipment malfunctions, etc.)
Such 'applied intelligence' is one of the areas where computers can literally extend the capacity and facility of human know-how. Expert systems and neural networks are two of the best examples to date. In effect, computers can learn how to do tasks, including deductive reasoning, that humans perform. Once learned, the execution of these tasks is normally much faster and more reliable than we ourselves can do them. Indeed, electronic computers were invented because human computers were too slow and error prone. Prior to the 1940s 'computer' was the name of a human occupation. Since then the methods of teaching electronic computers how-to-do have advanced from low- and high-level programming languages to extant methodologies such as knowledge engineering for expert systems and training of neural networks via practice and feedback.
Because computer systems contain praxic intelligence, we can teach them to do tasks. It is this very capability that makes it possible to use computers as an interactive medium for instruction and learning. It is interaction which sets computers systems apart from other media such as books, television and film. Computers can be taught how to interact with students in order to guide their learning. This is what makes their potential for transforming education so exciting (cf. Frick, 1991).
We must bear in mind, however, that computers literally do not understand the messages which they manipulate during these interactions. While computer systems exhibit praxic intelligence, we must not forget the distinction between it and qualitative and quantitative intelligence. Nor should we conflate praxic intelligence with inventive intelligence, which is discussed next.
Inventive intelligence is the ability to innovate and create. Intelligent natural systems can invent alternative ways of doing (e.g., to build a skyscraper, to catch a mouse). They can also envision and create entirely new ways of doing and their results. For example, the Wright brothers developed a vehicle that did not fly like any bird but like an airplane; Beethoven composed Symphony No. 9; and Leonardo da Vinci painted the Mona Lisa; Einstein created a theory of relativity.
Innovation is the design of new way to do something. When Thomas Edison invented the electric light bulb, it was a new way to provide light, compared to candles, kerosene lanterns, and gas lights. The end was the same -- to provide light -- but the means was different and novel. When the transistor was invented at Bell Laboratories, it was a different way to indicate an on/off state in an electronic circuit, compared to the previous use of vacuum tubes in computers and other electronic devices. Integrated circuits were yet another invention following the transistor that achieved the same end.
Maccia (1989) indicates that the following conditions must be met for innovative knowing:
S knows how to innovate the doing of P if and only if
1. S has the capacity for doing P.
2. S has the facility for doing P.
3. S smoothly executes constituents and
succession of
movements into some performance Pn when P
includes Pn , and
Pn is not equivalent to P.
4. P is a doing.
In creation the end is new. Writing this article is an example of creative know-how. The article itself did not exist before. The means of writing -- in this case my typing with a word processing program on a computer system -- is the same method that I use to write other things. The end is different and new. When Mozart wrote his Clarinet Concerto, this was also an example of creative know-how.
Maccia (1989) indicates that the following conditions must be met for creative knowing:
S knows how to create the doing P if and only if
1. S has the capacity for doing P.
2. S has the facility for doing P.
3. S smoothly executes constituents and
succession of
movements of P(1, 2, ... n) into P2 where
P(1, 2, ... n) are elements of P and P2 is not included in
P.
4. P is a doing.
Artificially intelligent computer systems have failed miserably in this domain, other than perhaps some creative computer art and music. However, such systems cannot tell if their compositions are any good because they lack the ability to appreciate (see 1.3 above).
When we write something new as I am now doing, inventive intelligence is required as well as praxic, qualitative and quantitative. Oftentimes the same is true for speaking. Writing something new about computers and instruction, for example, requires acquaintance with particular computers, as well as concrete experience with education and use of computers in education -- i.e., qualitative knowing. It requires adequate concepts about computers and about education -- i.e., quantitative knowing. The writing also requires praxic knowing -- how to set down the ideas, lay them out, organize the exposition of the possible relationships between computers and instruction. These three kinds of knowing are necessary but not sufficient. Still more is required to generate the ideas: inventive intelligence, the creation of something new.
Conversely, if we are simply copying something already written or literally reciting something, then this is not inventive intelligence, but praxic. To play Mozart's clarinet concerto is praxic, but to compose it was inventive. To follow a recipe to make a chocolate cake is praxic, but to develop a new recipe is inventive. To execute a computer program is praxic, but to devise the original program is inventive.
Furthermore, both Maccia and I believe that invention cannot be taught but only realized. Human teachers cannot directly teach students to innovate or create. Teachers themselves can innovate and create, serving as models. If students directly imitate or duplicate their teachers' inventions, then they are learning protocolic or conventional performances. This may be a useful step in order to prepare students for invention, but to exercise inventive intelligence a student must create something that is new to him or her.
Computers can nonetheless serve as powerful tools to facilitate human invention. For example, word processing programs have become a boon to writers who can type. CAD-CAM (Computer-Assisted Design and Manufacturing) programs have become indispensable to engineers. Interesting software has been developed as tools for artists and musicians to facilitate composition. And most significantly, authoring systems and authoring languages become powerful tools for instructional developers, with respect to development of computer-mediated instruction.
Instructional design and development requires inventive intelligence. Artificially intelligent tutoring systems would therefore require inventive intelligence in order to create instructional interactions and dialogues with students.
Qualitative intelligence is pivotal in Maccia's theory. Maccia (1987) concludes that four conditions are necessary for qualitative cognition: (1) the state of affairs which is the object of cognition must be epistemically present; (2) the presence may be perceptual or imagined, but the image must be accurate and complete; (3) the association between the knower and the known must be intimate or heightened; and (4) the object of cognition must be a unified whole with its own identity having characteristics that are discrete. (p. 217)
Without qualitative intelligence, the other kinds are not grounded in experience with the world around us. This in turn implies that tutoring systems, whether natural or artificial, must possess qualitative intelligence as well as quantitative and praxic; otherwise they blindly reason and follow procedures that manipulate symbol systems, images, sounds, icons and the like with no cognition of their meaning. Inventive intelligence is also a desirable property of a tutoring system, so it can realize new ways of doing and develop deeper understandings through disciplined inquiry.
Cognitive scientists are beginning to realize the epistemological significance of 'knowing that-one.' For example, the term, 'situated cognition,' has begun to appear in the literature in the last decade. Jerome Bruner (1990) reviews this evolution in cognitive psychology before discussing the current revolution, which he refers to as cultural psychology. Bruner views 'mind' as a creator of meanings, a special interaction through which it both constitutes and is constituted by culture. This is an entirely different paradigm than the view of mind as information processor, a view that was previously dominant in cognitive psychology.
I believe that we shall be able to interpret meanings and meaning-making in a principled manner only in the degree to which we are able to specify the structure and coherence of the larger contexts in which specific meanings are created and transmitted.... It simply will not do to reject the theoretical centrality of meaning for psychology on the grounds that it is "vague." Its vagueness was in the eye of yesterday's formalistic logician. We are beyond that now. (Bruner, 1990, pp. 64 - 65)
The larger context to which Bruner refers is culture. When we communicate we use 'signs.' 'Signs' are the "stuff" of communication which are embedded in our culture. 'Signs' are not limited to spoken and written words (linguistic symbols), but also include icons, gestures, pictorial representations, non-linguistic sound patterns, smells, touch, facial expressions, expression of emotion, demonstration by enactment, etc. 'Signs' take on meaning for humans through use during situated action.
It is true that computers can process 'signs' such as key presses and mouse clicks in the sense of decoding, storing, transmitting and encoding them. However, present-day computer systems cannot be said to understand the 'signs' they manipulate. The only 'signs' computers do appear to understand are the bit patterns which represent machine language instructions. The meaning of each machine language instruction is unambiguously prescribed. Each is associated with a primitive action the computer can do, such as move a collection of bits from one place to another, and perform arithmetic and logical (Boolean) operations on those bit collections. While a collection of bits may be an encoded representation of external 'signs', a computer is literally isolated from the culture in which those 'signs' are given meaning. Computer systems are blind and deaf to the symbol systems, images, icons, sounds, etc. that they process.
For example, the word-processing program I am now using has no idea whatsoever what I am typing. It merely processes the symbols I strike on the keyboard and follows my directions for editing. It does not know what I mean by the words I choose for expressing my ideas. On the other hand, human readers can understand my message which a computer has printed so nicely on paper.
The same is true for computer-mediated learning products, whether they are guided practice, tutorials, simulations, interactive video, games, hypertext, or multimedia. A computer merely carries out the directives of a human tutor who originally designed the particular instructional or informational system and its subject matter. The computer system has no idea of the meanings of the messages (i.e., groups of 'signs') being sent back and forth between the tutor and students. The computer system is a medium which conveys those human messages.
If Maccia is right, then current efforts to design artificially intelligent tutoring systems are doomed to fail unless they can satisfactorily address the grounding problem -- grounding that occurs through qualitative cognition. That is indeed a very significant implication. From Bruner's perspective, such systems will never learn the meaning of the 'signs' they manipulate unless they become truly interactive with the culture in which the signs are embedded. In other words, computer systems will somehow need to "live in" and experience the culture with us. This appears to be the very same grounding that concerned Maccia (1987) in his discussion of genetic epistemology of natural intelligence (Note 4). It will be interesting to see if Maccia's and Bruner's theories are upheld or refuted by further inquiry.
Artificially intelligent tutoring systems outside of the praxic domain appear to be beyond our reach at this time. This does not mean that a computer cannot be used to facilitate teaching-learning processes. Instead, it means that it is a medium for conveying representations of what teachers and students know, believe, feel and intend.
In the closing section of Restructuring Education through Technology (Frick, 1991), I reiterated what the educational philosopher, Elizabeth Steiner (1981), so cogently argued: The primary role of a teacher is to select the best of culture and make that culture available to students in their learning environment. Computers cannot do that, nor should they be doing the selecting. Nonetheless, computers like television are part of our culture. Significant amounts of content can be made available through these media, just as it has been made available through textbooks and other print media. Teachers and other educators should be selecting the best of culture. To paraphrase a Will Rogers' comment about horses and water: Teachers can lead students to culture, but they can't make them drink it.
Following Plato, the major purpose of education is to enhance the quality of life. Through selection of the good, the true, and the beautiful, and by helping our younger generations to form relationships with such content, humankind may have a chance to evolve in a positive direction.
Note 1.
See The Machine that Changed the World. Part 4, "The Thinking Machine," provides an excellent discussion of the challenges of knowledge representation and natural language understanding. Numerous scholars are interviewed on this videotape including Marvin Minsky, Edward Feigenbaum and Hubert Dreyfus. The Turing Test is also well illustrated.
Note 2.
Maccia (1988) used the terms procedural and performative intelligence to characterize ability to know how. I refer to this domain as praxic, a coined adjective based on Aristotle's notion of praxis. Maccia described innovation and creation as further kinds of know- how, to which I refer collectively as inventive intelligence.
Note 3.
In addition to cognitive learning outcomes, there are also affective and conative outcomes (Steiner, 1988). Students can develop sensitivities and intentions, respectively. For example, to develop a love of archaeology and want to become an archaeologist is as important, if not more so, than knowing something about archaeology.
Note 4.
Considerable discussion of Maccia's epistemology was necessary in this article, since it is apparently not well-known at this time -- compared to Howard Gardner's notions. The basic difference is that Gardner appears to be making distinctions among abilities to come to know classes of objects of knowing, such as linguistic symbol systems, music, mathematics and logic, interpersonal relationships, and so forth. On the other hand, Maccia is characterizing intelligence from an epistemological perspective. The two theories are complementary. For example, one can exercise qualitative, quantitative, praxic and inventive intelligence with music, with linguistic symbol systems, with kinesthetic arts, and the like. Furthermore, cross-classifications between these theories provides us with a new structural view of curriculum in education. For example, it would be potentially useful to think of learning activities in mathematics and logic in terms of Maccia's twelve kinds of knowing. Finally, and perhaps most important, Maccia has provided us with criteria for assessing student learning for each of these twelve kinds of cognition.
Arbib, M. (1977). Computers and the cybernetic society. New York: Academic Press.
Aristotle. (1973). Introduction to Aristotle (2nd edition, edited by Richard McKeon). Chicago: University of Chicago Press.
Asimov, I. (1964). The rest of the robots. Garden City, NY: Doubleday.
Asimov, I. (1985). Robots and empire. New York, NY: Ballantine Books.
Bateson, G. (1979). Mind and nature. New York, NY: E. P. Dutton.
Bruner, J. (1966). Towards a theory of instruction. Cambridge, MA: Belknap Press of Harvard University.
Bruner, J. (1990). Acts of meaning. Cambridge, MA: Harvard University Press.
Clarke, A. (1968). 2001: A space odyssey. New York, NY: The New American Library.
Davis, R. (1984). Amplifying expertise with expert systems. In P. Winston and K. Prendergast (Eds.), The AI business: The commercial uses of artificial intelligence. Cambridge, MA: The MIT Press, 17- 40.
Dewey, J. (1916). Democracy and education: An introduction to the philosophy of education. New York, NY: The Free Press.
Dreyfus, H. (1979). What computers can't do: The limits of artificial intelligence (revised edition). New York, NY: Harper and Row.
Dreyfus, H. (1992). What computers still can't do: A critique of artificial reason. Cambridge, MA: MIT Press.
Duffy, T. and Jonassen, D. (1992). Constructivism and the technology of instruction. Hillsdale, NJ: Lawrence Erlbaum.
Einstein, A. (1961). Relativity. The special and general theory. New York: Crown Publishers.
Frick, T. (1991). Restructuring education through technology. Bloomington, IN: Phi Delta Kappa Educational Foundation.
Frick, T. (1992). Computerized adaptive mastery tests as expert systems. Journal of Educational Computing Research, 8(2), 187-213.
Frick, T., Miller, B., Wise, A., Bray, M., Ahmed, M., Chao, C.-C., Nelson, L., Sweeney, M. and Whyte, G. (February, 1995). Hypermedia on the Internet: Development of a World-Wide Web server. Paper presented at the annual meeting of the Association for Educational Communications and Technology.
Frick, T. & Reigeluth, C. (Feb., 1992). Verifying Instructional Theory by Analysis of Patterns in Time. Paper presented at the annual conference of the Association for Educational Communication and Technology, Washington, DC.
Gagne, R. (1985). The conditions of learning and theories of instruction (4th Edition). New York, NY: Holt, Rinehart and Winston.
Gardner, H. (1985). Frames of mind: The theory of multiple intelligences. New York, NY: Basic Books.
Kay, A. (1990). User Interface: A Personal View. In Laurel, B., The art of human-computer interface design. Reading, MA: Addison- Wesley, 191-207.
Kearsley, G. (1987). Artificial intelligence and instruction. Reading, MA: Addison-Wesley.
Leshin, C., Pollock, J., and Reigeluth, C. (1992). Instructional design strategies and tactics. Englewood Cliffs, NJ: Educational Technology Publications.
Maccia, G. (1987). Genetic epistemology of intelligent natural systems. Systems Research, 4(1), 213-218.
Maccia, G. (1988). Genetic epistemology of intelligent systems: Propositional, procedural and performative intelligence. Paper presented at the Fourth International Conference on Systems Research, Cybernetics and Informatics, Baden-Baden, West Germany, August.
Maccia, G. (1989). Personal communication and unpublished manuscript.
McKeown, P. (1993). Living with computers (4th Edition). Fort Worth, TX: Dryden Press.
McLuhan, M. (1964). Understanding media: The extensions of man. New York, NY: McGraw-Hill.
Merrill, D. (1983). Component display theory. In C. M. Reigeluth (Ed.), Instructional-design theories and models: An overview of their current status. Hillsdale, NJ: Lawrence Erlbaum Associates, 279-334.
Merrill, D. (Feb., 1994). Automating instructional design: Recent advances in instructional transaction theory and ID expert. Paper presented at the 1994 conference of the Association for Educational Communications and Technology (AECT), Nashville, TN.
Montessori, M. (1965). Spontaneous activity in education. New York, NY: Schocken Books.
Neapolitan, R. (1990). Probabilistic reasoning in expert systems. New York, NY: Wiley.
Plato. (1945). The republic of Plato (translated with introduction and notes by F. M. Cornford). London: Oxford University Press.
Peirce, C. S. (1977). Complete published works. Greenwich, CT: Johnson Associates.
Sleeman, D. and Brown, J. (Eds.) (1982). Intelligent tutoring systems. New York, NY: Academic Press.
Steiner, E. (1978). Logical and conceptual analytic techniques for educational researchers. Washington, DC: University Press of America.
Steiner, E. (1981). Educology of the free. New York: Philosophical Library.
Steiner, E. (1988). Methodology of theory building. Sydney: Educology Research Associates.
The machine that changed the world (Video series, 1993). Princeton, NJ: Films for the Humanities & Sciences.
Winston, P. and Prendergast, M. (1984). The commercial uses of artificial intelligence. Cambridge, MA: MIT Press.
Last updated by T. W. Frick, Nov. 20, 2023