4yG:@((vA.B A Methodology for the development of 'Intelligent' Educational Systems Dick J. Bierman & Paul A. Kamsteeg University of Amsterdam Abstract The research project "A computer coach for thermodynamics" aimed at building a tutoring system around an existing computer model of problem solving in thermodynamics. The resulting prototype tutoring system exists of four modules: a domain expert, a diagnoser, a tutoring expert and a graphic interface. Problems surfacing during the project, partly due to the difficulty of eliciting specific didactic knowledge, lead to the development of a method (MUSPA) of knowledge elicitation for use in an interactive computer system, using the developing system itself. A pupil works with the incomplete system, while at another terminal a human coach simulates the tasks the system cannot (yet) perform. Both pupil and human coach think aloud and also their written interaction is recorded. The method consists of three stages: construction of the interface, construction of a minimal prototype, and refinement and extension of the prototype. LEVEL1 Introduction In the research project 'Knowledge Acquisition in formal Domains' the main topic is how people solve problems e.g. in the domain of physics. The research method is based upon the analysis of the protocol produced by a subject if the subject is asked to 'think aloud' while solving a problem. It has been argued that such protocols would reflect the mental processes reasonably undistortedly (Breuker et al, 1986). The analysis of such protocols should result in a model of the problem solving behavior in the respective domain, which in our case is the domain of thermodynamics. Preferably the model should be flexible enough to incorporate several types of problem solving behavior such as those of experts but also those of novices. In practice this approach implies a continuous cycle of analysis and model construction, where protocols are analyzed within the framework of the current model and where differences cause further refinement and adaptation of the model. Models are implemented as computer programs using Artificial Intelligence techniques. Evidently, a computer program which claims to be a model of an advanced probem solver should be able to solve the problems too. And the 'mental trace' of such a program should correspond with the trace of the human problem solver in as far as this can be deduced from the protocol. In the same vein a good model of a novice problem solver should make the same errors as its human counterpart. Interestingly, this methodology enables cognitive research of single subjects. And because evaluation of this type of AI-based models is not possible in a quantitative way, indeed only one crucial test for such a model appears to be feasible: to construct the model on the basis of many protocols of the same subject and then compare the behavior of that single subject and the model on a specific new problem. They should solve this new problem in an identical way. However, in practice, models are based upon the protocols of many subjects. The model thereby gains in generality but looses in specificity and falsifyability. In the early 80's we had developed within our project a working model for an advanced solver of simple problems in the domain of +FootnootStart+ Rather a very obedient novice who is solving the problems exactly according to the rules. +FootnootEinde+ thermodynamics. A typical example of problems it is able to +FootnootStart+ Actually a subset of thermodynamics, concerning only the very first principles of Heat theory. +FootnootEinde+ solve is the following: A container which is closed with a piston contains an ideal gas. The volume is 2 liters and the pressure is 120 kPa. By moving the piston slowly outwards the volume is increased to 3 liters. The temperature is kept constant. What is the pressure now? Apart from our interest in problem solving itself, we are also interested in the question how people learn to solve problems. For instance, recently a model has been constructed for a novice problem solver (Jansweijer et al, 1986). The development of this model into a model of the advanced problem solver would in itself be a model for learning of this capacity. Another way to gain insight in the learning of problem solving capacities is to study the effect of different educational programs. One could, on the basis of a model of learning, construct optimal educational interventions. If these are more successful than others one might conclude that the model of learning is correct. Anderson, for instance (Anderson 1987) claims that the use of knowledge based educational systems as instrument for cognitive research is very promising indeed. As part of the aforementioned research program 'Knowledge Acquisition in formal domains' we have recently finished the construction of such a knowledge based computer coach which in principle will enable us to manipulate different educational strategies. Tutorial interventions of the computer coach should be based on an explicit dynamically adjusted model of the cognitive state of the pupil. The computer coach could be seen as a model of an ideal individual teacher. +FootnootStart+ Ideal, in the sense of being completely consistent and having a perfect memory. +FootnootEinde+ Since a teacher in general is an advanced problem solver by profession the computer coach should be able to solve the problems too. As mentioned before at the start of this computer coach project we already had a system that could solve problems in a psychological plausible way. So it seemed reasonable to expect that the implementation of other parts of the system would not cause too much problems. This expectation turned out to be overly optimistic. What actually happened was, that we had to devise a new method of knowledge elicitation in order to be able to proceed with the project. This knowledge elicitation method can be considered to be an important product of the project since it opened a new avenue in research into teachers thinking. LEVEL1 The computer coach for thermodynamics The functional structure (the architecture) of the computer coach, as it relates to solving one single problem, is sketched in fig.1. +FootnootStart+ On top of this architecture there is a component which either selects the next problem or decides that the pupil has learned enough. +FootnootEinde+ The system consists of four components. Each of these components performs a separate function. n+CenterStart+ fig.1: the architecture of the computer coach. +CenterEnd+ LEVEL2 The domain expert This is the original 'advanced problem solver'. This component accepts a problem text as input; consecutively it solves the problem, producing a 'mental trace'. This trace is also called 'norm' trace because of the underlying educational goal that the pupil should eventually confirm to this solving norm. +FootnootStart+ Of course, the idea that there is one single norm is nonsense. For instance, the 'mental trace' of the computer contains sequences which can be permuted. More generally put: a norm trace represents only one program of actions, whereas there may be several correct alternatives. +FootnootEinde+ From the beginning of the project on, the intention has been to localize all knowledge pertaining to the domain of thermodynamics within this component. The reason being that a similar computer coach for a different domain could then be constructed by replacing nothing but the domain expert. LEVEL2 The diagnose expert The function of the diagnose expert is to generate hypotheses about the reason for deviating cognitive behavior on the part of the pupil. A further important task is to update a model of the pupil. One way of doing this is estimating, for each knowledge item that the domain expert has access to (factual knowledge, procedural knowledge as well as knowledge about solve strategies), the mastery level of the pupil. In this approach the current knowledge of the pupil is seen as a subset of the eventual knowledge he should come to possess. The pupil model is therefore an incomplete, but apart from that identical, copy of the expert model. This kind of pupil model is called 'overlay model', because (apart from the missing elements) it exactly covers, as it were, the expert model. In domains, such as physics, about which one may assume that a pupil already has some (possibly incorrect) knowledge to begin with, it is furthermore necessary to keep track of the incorrect knowledge (misconceptions, incorrect mental models) the pupil has. Since this is knowledge an expert generally does not have, it does not fit in an 'overlay model' but must be represented separately, for instance in a list of misconceptions ('bug catalog') or by extending the overlay model with pointers to incorrect knowledge. The aforementioned function of generating hypotheses about deviating pupil behavior is performed in the context of such a pupil model. That is, the likelihood of a certain hypothesis is deduced on the basis of information in the pupil model. LEVEL2 The tutoring expert The task of the tutoring expert is to decide upon an optimal instructional intervention with respect to the diagnosis. In this task also, the pupil model plays an important role. Apart from the cognitive aspects of the pupil model, personality features of the pupil can be taken into consideration in deciding upon an optimal instructional action. E.g. the personality trait 'negative fear of failure' can play a part in deciding whether to give either much or little procedural hints. LEVEL2 The interface An (user) interface is that part of a computer program which takes care of the interaction between the user and the rest of the program. In our case this means that actions the pupil performs at the computer terminal have to be translated into a form that can be used by the diagnosis expert. That is, motor actions have to be interpreted as reflections of cognitive actions. Reversely, instructional interventions of the tutoring expert have to be presented to the pupil in the form of one or more comprehensible sentences. LEVEL1 The results of the computer coach project Firstly, the project has spawn a prototype of a so-called 'intelligent' computer assisted instruction system. The intelligence of the system appears from a twosome of aspects. The system is capable of understanding the pupil conduct; that is, it can interpret pupil behavior in terms of its own actions. +FootnootStart+ Real intelligence should manifest itself in the capability of REPRODUCING pupil behavior, e.g. by corrupting a part of the internal knowledge. However, this would quickly lead to combinatorial explosion; therefore this goal has not been aimed for in the current project. +FootnootEinde+ Furthermore, the system continuously infers new knowledge, specifically with regard to a cognitively substantial pupil model, by using agreements and deviations between its own actions and those of the pupil. Secondly, and this is in our opinion more important, a substantial number of problems has come to light during the work on the project. For some problems, specifically those of a technological nature, a solution has been found. Concerning the problems of a fundamental nature, a methodology has been designed by which we expect to eventually solve them. LEVEL2 Problems of a technological nature Part of these problems concerns translating the motor behavior of the pupil at the terminal into cognitive actions. This part has been solved by on the one hand not allowing the pupil to use any paper while working at the terminal, and on the other hand extending the interface with an interpretable 'electronic scratch pad'. This is a section of the videoscreen on which the pupil may type in text (givens, etc.), copy formulae from a database, and assemble drawings from preconstructed parts (Bierman & Anjewierden 1986; Bierman & Kamsteeg 1986). Furthermore, the structure of the domain expert system which was available at the start of the project appeared be in some aspects ill-fitted for smooth inclusion in the total system, specifically with regard to explicating the precise knowledge used in a solve step (Kamsteeg & Bierman 1987). LEVEL2 Problems of a fundamental nature In the design of knowledge based systems three types of knowledge may be discerned. General world knowledge ('common sense knowledge') appears to play a role in solving problems in nearly every domain. This role is most explicit in translating the raw problem text into a representation which is adequate for the relevant domain. E.g. the knowledge concerning the characteristics of objects like balloons or bicycle-pumps, which may feature in a problem text. Global domain knowledge is commonly known and usually of an algorithmic nature. Examples are definitions of and relations between concepts, fixed procedures, etc. One can elicit this type of knowledge through interviews with experts or directly from text books. Specialist's (empirical or intuitive) knowledge is of a more heuristic nature and is seldomly found in written form; usually it appears to be difficult to elicit by post hoc interviews as well. A much used technique to get at this type of knowledge is by asking experts to solve problems while thinking aloud. In the context of expert system development a special methodology for knowledge elicitation has even been designed (Wielinga & Breuker 1984). Likewise, the expertise relating to individual tutoring is nowhere described in full. Even global knowledge about this subject is hard to find (Kamsteeg 1984). Educational theories are usually phrased in global terms and are hardly operationalized in terms of teacher actions (Knoers 1973). Furthermore, most theories implicitly assume a context of classroom education. Hence, from the viewpoint of individual coaching, they overemphasize organizing and managing tasks, and underemphasize individual diagnosis and interaction. An exemplary quote in this respect is the following, by Ausubel, Novak & Hanesian (1978,50): ... the teacher's most important and distinctive role in the classroom is still that of director of learning activities.... In fact, barring a single exception (Mettes & Pilot 1981; Mettes & Roossink 1982), there is no theory of education in problem solving, although several authors express the need for theories of this kind (Reif 1980; Shuell 1980). An additional difficulty in this area is, that specialist's knowledge of an individual teacher cannot be elicited directly through the method of think-aloud protocol analysis, since the teacher thinking aloud would interfere drastically with the individual teaching he is engaged in simultaneously. LEVEL1 The MUSPA method for knowledge elicitation The Multiple Source Protocol Analysis (MUSPA) method is actually the combination of two well-known traditional methods. It is not uncommon for system developers to confront 'the' future user of the system with a pseudo-version in an early stage of the implementation (generally as early as the specification stage). The 'intelligence' of the pseudo system is supplied by a human at another terminal who simulates the responses of the to-be-developed system. This technique is known as the 'Wizard of Oz' method after an American tale. In that tale the main characters (users) belief to interact with a real wizard (computer system) but it turns out that this wizard is just an ordinary person. Within the MUSPA approach this 'Oz' method is combined with recording of thinking aloud protocols of the person who simulates the system (in our case an expert teacher) and the user (in our case the pupil). Consequently three protocols are obtained, which are merged into a single protocol of the session (hence the name Multiple source protocol). The study of such protocols provides information with regard to the diagnostic and tutorial knowledge used by the human teacher. As a bonus it provides information about the filtering through the interface which might cause misunderstandings +FootnootStart+ Since the teacher and the pupil communicate through the computer, certain communication methods (such as facial expressions) are impossible. In this respect the interface functions as a filter which only passes part of the information. +FootnootEinde+ between the teacher and the pupil separated by this narrow bandwidth communication channel. A fundamental aspect of this approach is the sequencing in three discrete stages. Each stage results in the elicitation of knowledge which is implemented in the system in the next stage. At the final stage there is enough knowledge to have the system function independently from human help. In that stage experimental sessions will result in refinement of the knowledge base. Thus the educational task, initially performed by the human teacher is gradually taken over by the system. The computer coach is therefore an essential instrument in the process of its own creation. During this gradual development of the system the teacher's task shifts from thinking aloud to 'criticizing aloud'. LEVEL2 MUSPA- stage 1: The computer coach does not exist at all In the initial stage the teacher and the pupil communicate through the alphanumeric terminals (fig.2). n+CenterStart+ fig. 2: MUSPA stage 1 +CenterEnd+ The pupil performs the actual problem solving using paper and pencil. Paper copies of the set of problem texts and of a list of hints are provided. A typical protocol fragment is given in fig.3. TABELSTART cS cS cS cS cS lS. = TIME SOURCE INTERACTION - 03.30 T: OK, YOU MAY START WITH PROBLEM 5 05.00 T: COULD YOU TELL ME ONCE IN A WHILE WHAT YOU ARE DOING OR WHEN YOU NEED HELP? 06.00 P: COULD YOU GIVE ME THE FORMULA OF THE MOL? 07.30 T: YOU'VE GOT SOME HINTS ON PAPER. DID YOU ALREADY NOTE WHAT THE 'SYSTEM' IS? 09.00 P: WHAT KIND OF 'SYSTEM' DO YOU MEAN? 10.00 T: HAVE ANOTHER GOOD LOOK AT THE HINTS AND EXECUTE HINT 1 AND 2. WHEN YOU'RE READY PLEASE SAY SO. 12.30 P: I FINISHED THAT 12.31 T: WHAT KIND OF OBJECT DID YOU DRAW? - lr s s cR s s lr s s lr s s. fig.3: MUSPA - stage 1, protocol fragment (only interaction part shown) +EindeTabel+ Each line of the protocol starts with a number indicating the time after the start of the session. The source of the interaction is indicated with a T(eacher) or a P(upil). Note that in this protocol the teacher needs 3.5 minutes to select a proper problem. From the thinking aloud protocol it appears that this delay is due to the fact that the teacher doesn't know anything about the pupil. His considerations to choose problem 5 reflect didactic principles which might be implemented in the next stage of the development of the computer coach. A recurrent theme in the protocols produced in this stage is the teacher's lack of information with regard to what the pupil is doing. +FootnootStart+ Because of this lack of information the teacher was forced to probe the pupil rather extensively. These probes sometimes unwittingly function as hints. +FootnootEinde+ Teachers especially were frustrated not to know what notes and sketches the pupil was making. On the basis of these findings a number of pupil-machine interface criteria were formulated, the most important being that the pupil should be able to make notes and sketches in a natural way on the machine itself. After implementation of this interface (the 'Electronic Scratchpad') the second stage of the MUSPA-sessions started. LEVEL2 MUSPA-stage 2: The interface is implemented In this stage there is no educational knowledge implemented but the interface is ready. Thus the pupil experiences the system in it's intended final form. Next to the alphanumeric communication terminal the teacher now has a graphic terminal which enables him to view the graphic manipulations of the pupil on the 'electronic scratchpad' (fig.4). A typical protocol from this stage is given in fig.5. n+CenterStart+ fig.4: MUSPA stage 2, experimental situation with implemented interface. +CenterEnd+ TABELSTART cS cS cS cS cS lS. = TIME SOURCE INTERACTION - 02.09 P: TO THIS END I WILL USE THE FIRST LAW 02.38 T: WOULDN'T YOU FIRST ANALYZE THE SITUATION? 03.33 P: THE PRESSURE CHANGES THE TEMPERATURE IS CONSTANT THE VOLUME CHANGES 03.54 T: I ALSO MEAN A DRAWING - lr s s cR s s lr s s lr s s. fig.5: MUSPA - stage 2, protocol fragment (only interaction part shown) +EindeTabel+ An analysis of this and other similar protocols shows a fair amount of consistency in the intervention strategy of the teacher. When the pupil deviates from the ideal (norm) path, a sequence of increasingly specific hints appears to be given. E.g. in this protocol the teacher first asks Wouldn't you first analyze the problem? (before jumping to selection of a formula) and later on, when this hint didn't have the right effect, the teacher suggests more explicitly I also mean a drawing . Consequently a hint generator was implemented which embedded this kind of educational strategic knowledge. In the same way a superficial diagnoser was constructed. All modules of the computer coach now had sufficient knowledge to work alone without having to rely on knowledge provided by the human coach. Although the knowledge was sufficient to have a stand-alone computer coach it was still very superficial. This initiated stage 3. LEVEL2 MUSPA-stage 3: The computer coach is born. The sessions in this stage are intended to elicit knowledge to refine the implemented knowledge. This is done by asking the teacher to pay attention to considerations and actions of the system that he doesn't agree with. In other words he is requested to criticize the system. To this end the teacher is enabled to 'overrule' the system's proposed tutorial actions. In order to enable this task the internal 'thinking' of the computer coach is shown on the terminal (fig.6). n+CenterStart+ fig.6: MUSPA stage 3, the trace screen of the teacher. There are three windows, representing the functioning of the Interface, the Diagnosis and the Tutorial modules respectively. Apart from this information the terminal indicates which part of the norm-trace is matched against the actual pupil behavior. The cognitive steps of the system are represented in the form of IF (condition) THEN (action) rules. Both the conditions and the actions are shown on the teacher screen. If the tutoring component decides to perform an interaction, this proposed action is shown to the teacher and the system halts to offer the teacher the opportunity to modify, cancel or replace this action with his own intervention. The interaction protocol in this stage 3 therefore has three participants. the pupil, the human coach and the computer coach. A typical example is given in fig.7. In order to comprehend what is going on we have added in this protocol information about the pupil's solve actions (between square brackets). TABELSTART cS cS cS cS cS lS cS cS lS cS cS lS cS cS lS cS cS lS cS lS s cS cS lS cS cS lS cS lS s cS cS lS cS cS lS cS lS s cS cS lS cS cS lS cS lS s cS cS lS cS cS lS cS lS s cS cS lS cS lS s cS cS lS cS cS lS cS lS s cS cS lS cS lS s cS cS lS. = TIME SOURCE INTERACTION - 00.19 S: HELLO....HERE IS YOUR (HYBRID) TEACHER. LET'S START! 01.51 S: SEARCHING FOR SUITABLE PROBLEM. ONE MOMENT PLEASE... 02.08 S: HERE COMES YOUR PROBLEM. GOOD LUCK. [pupil makes drawing containing trivial error] 11.55 T: OK GO AHEAD <> ["V1 is 2 liter"] 17.41 S: OK, BUT YOU DIDN'T FINISH THE PROBLEM YET: YOU SHOULD FINISH THE PROBLEM SKETCH. ["P1 is 120 kPa"] 21.22 S: OK, BUT YOU DIDN'T FINISH THE PROBLEM SKETCH YET: YOU SHOULD WRITE DOWN THE PROBLEM FEATURES. ["T1 is T2"] 26.09 T: GO AHEAD <> ["closed system"] 27.32 S: YES, GO AHEAD ["V2 = 3 liter"] 29.44 T: GO AHEAD <> ["pressure diminishes"] 31.44 S: YES, GO AHEAD [pupil chooses formula: Boyle's law] 35.29 T: IS THIS FORMULA VALID FOR ALL GASES? 37.13 P: NO, ONLY FOR IDEAL GASES, SO IT'S OK TO USE IT HERE 38.44 T: LOOK AT THE PROBLEM AND EXTEND YOUR PROBLEM SKETCH WITH THESE KIND OF FEATURES <> - lr s s cR s s lr s s lr s s. fig.7: MUSPA - stage 3, protocol fragment (only interaction part shown) +EindeTabel+ Note that the human coach decides to give a positive reinforcement (OK, go ahead) and to this end rejects the null-action proposed by the system. Also it can be observed that the human coach gives an (implicit) hint about the meaning of the phrase "problem feature". The computer coach is not able to generate this hint from its present knowledge base. Thus further knowledge should be inserted to incorporate similar interventions. LEVEL1 Expectations and conclusions A knowledge based CAI system is a large and complex program, especially if the educational knowledge concerns a semantically rich domain like physics. Problem solving in this domain requires complex cognitive skills and it is difficult to build a good computer coach which also has reasonable response rates. We don't expect that such a coach will be realized before the end of this decade. Even our prototypical system, which is certainly still lacking deep educational expertise, works only on relatively expensive systems like a VAX-computer. And the response rates, in the order of minutes, are only acceptable in an experimental setting. But the price/performance ratio of hardware is decreasing orders of magnitude every 10 years and it is certainly not impossible that PC's with the required power are available for use in a practical setting like the classroom before the turn of the century. One should not forget that the present computer coach has been developed with a research goal in mind. The system is more geared towards flexibility (Prolog) than towards efficiency. However the major problem in progress is that we still have no deep insight in the process of individual teaching. We lack knowledge about the correct and incorrect cognitive structures which might be present in the pupil (pupil model). We don't know how to infer these structures from the pupil's actions at the terminal (Diagnosis) nor do we know which feedback or intervention, in which context and for which pupil has the optimal didactic effect (didactic knowledge). In order to build a computer based educational system, we first have to learn how to teach. Empirical research on the issue of expert-teacher thinking during individual coaching is mandatory (Ohlsson, 1986). The MUSPA methodology appears to be a valuable tool, not only in the context of educational research, but more generally for the elicitation of knowledge in the context of an ongoing dialogue. This dialogue is not disturbed by the thinking aloud of the participants. Furthermore the cyclical approach with an ever increasing involvement of the computer system, offers an increasingly better interpretation model of this dialogue and its underlying knowledge. Finally, the confrontation of the human teacher with this computer coach, which is to some extent a model of himself, offers a natural way to elicit criticism (or reflection) of the already implemented parts of his own knowledge. The whole cyclical approach appears to be the qualitative equivalent of the empirical cycle (De Groot, 1970) which is well known from the empirical (quantitative) sciences. References .Lt Anderson, J.R., Methodologies for studying human knowledge. In: Behavioral and Brain Sciences (yet to appear). Abstract in: UUCP newsgroup mod.ai, Message-ID: <8701071550.AA01452....>, 1987. .Lt Ausubel, D.P., J.D. Novak & H. Hanesian, Educational psychology, a cognitive view. New York: Holt, Rinehart & Winston, 1978 (2nd ed). .Lt Bierman, D.J. & A.A. Anjwierden, The use of a graphic scratchpad for students in ICAI. Proceedings of the 27th ADCIS Conference (pp.68-71), New Orleans, 1986. .Lt Bierman, D.J. & P.A. Kamsteeg, Het monitoren van het Gewenst Handelingsverloop met behulp van een "intelligente" computercoach (monitoring the preferred program of actions and methods using an "intelligent" computer coach). In: P. Verhagen & B.J. Wielinga (Eds.), Media in het onderwijs. Lisse(Neth): Swets & Zeitlinger, 1986. .Lt Breuker, J.A., J.J. Elshout, M.W. van Someren & B.J. Wielinga, Hardopdenken en protokolanalyse (thinking aloud and protocol analysis). In: Tijdschrijft voor Onderwijsresearch, 11/5, 1986, 241-254. .Lt Jansweijer, W.N.H., J.J. Elshout & B.J. Wielinga, The expertise of novice problem solvers. Proceedings of the European Conference on AI (pp.576-585), Brighton, 1986. .Lt Jansweijer, W.N.H., L. Konst, J.J. Elshout & B.J. Wielinga, PDP: A protocol diagnostic program for solving problems in physics. Proceedings of the European Conference on AI (pp.278-280), Paris, 1982. .Lt Groot, A.D. de, Methodology. The Hague(Neth): Mouton, 1970. .Lt Kamsteeg, P.A., Kennis van docenten bij individuele coaching (knowledge of teachers in individual coaching). Dept. of psychonomics memo 25.6.84.421. Amsterdam(Neth): University of Amsterdam, 1984. .Lt Kamsteeg, P.A. & D.J. Bierman, Constraints on an expert system for use in ICAI. Paper to be presented at 2nd World Basque Congress: conference on Artificial Intelligence, sept.7-11 1987. .Lt Knoers, A.M.P., Instructiemethoden (instructional methods). In: J.A. van Kemenade (Ed.), Bijdragen uit de onderwijswetenschappen. Alphen aan den Rijn(Neth), Samsom, 1973. .Lt Mettes, C.T.C.W. & A. Pilot, Linking factual and procedural knowledge in solving science problems: A case study in a thermodynamics course. In: Instructional Science, 10, 1981, 33-361. OC-rapport 42. Enschede(Neth): University of Twente, 1980. .Lt Mettes, C.T.C.W. & H.J. Roossink, Terugkoppelen bij het maken van vraagstukken (feedback for problem solving). OC-rapport 48. Enschede(Neth): University of Twente, 1982. .Lt Ohlsson, S., Some principles of intelligent tutoring. In: Instructional Science, 14/3-4, 1986, 293-326. .Lt Reif, F., Theoretical and educational concerns with problem solving: bridging the gap with human cognitive engineering. In: D.T. Tuma & F. Reif (Ed.), Problem solving and education, issues in teaching and research. Hillsdale, Erlbaum, 1980. .Lt Shuell, T.J., Learning theory, instructional theory and education. In: R.E. Snow, P.A. Federico & W.E. Montague (Ed.), Aptitude, learning and instruction, volume II: cognitive process analyses of learning and problem solving. Hillsdale, Erlbaum, 1980. .Lt Wielinga, B.J. & J.A. Breuker, Interpretation of verbal data for knowledge acquisition. Proceedings of the European Conference on AI (pp.41-50), Pisa, 1984. .Le A T KIND OF OBJECT DID YOU DRAW? uit G8;C br.HBpztzozizozozozozozdz@$p 04D[kzuzuzuzuzuzluzuzuzuzk""$$&&((*R*b+r+--151<8j8:zuzpzuzuzuzuzuzuzuzpz@:;@5@EA2ABDVD]EEEE,E7EFHGHLHHKLK\zuzuzuzuzluzuzuluzuzK\KLMMOOOOOOOQQQaQnRRSBSXzuzuzulzuzuzlzuzgzg@SXUUW'W6W;[-[8`` `S``bbllmmnn:zuzulzuzlzuzuzgzgzg@n:oop`pppqqr'r2rhrsAsqt;tbu"u<uuvtvzuzuzuzuzuzuzuzuzuzuzu@vw>w|xx}xxyAyByCyDyFyGyJyKyLyMyNynyoypzuzuzuzppzjzzzzzzz@@HI:;<QR3 uGDvmmmmmv^^vvvvvvvvv7hhhvvvvvvvvvvvvvvvvvvvvvvh!#$$$$&&&&(()) ,T----0 vvvvvvvvvvvvvvvvvvvvvvh0 14151^1_13/4759::;-;.=ADUDVDDEEEvvvvvvvvvvvvvvvvvvvvvvhEEEEE+E,E8E9JMMMMOzO{O|O}O~OOOOOvvvvvvvvvvvvvvvvvvvvvvhOOUUUDUEW"W#W$W%W&W'WmWnbbb8b9illllvvvvvvvvvvvvvvvvvvvvvvhllllyAyByCyDyEyFyGyKvvvvJFJFFv;  h+# p @P !h 'quotpunt1+@s7!@" @RR!8Dh7hxGyGpk:K\SXn:vyJ0 EOlyGypyrysytyuyvywyxyyyzy{y|zuuuuuuuuu yKyLyMyNynyoypyqyrtgTCTC::hh h h h   h  hyrysytyuyvywyxyyyzy{jWFWFF333hn hn  hn   hn y{y|{vyGpk:K\SXn:vypy|0 EOlyKyry{y|  h  h n DDDDEEEFGGGQGRIuIvYYYYY]]]]]]]]]__vvvv?yJE?yKF?yL?yMF9?yN?ynHF?yoHJ€J?yp\'?yr?ys\A?yt?yu?yv?yw`?yx?yy?yz``bby{yAyF