THE EFFECTIVENESS OF AN INTERACTIVE MAP DISPLAY IN TUTORING GEOGRAPHY

Transcription

1 a BOLT B E R A N E K AND NEWMAN 'NC a W- CONSUTNG DEVEOPMENT RESEARCH / BBN REPORT NO <^y to o THE EFFECTVENESS OF AN NTERACTVE MAP DSPLAY N TUTORNG GEOGRAPHY CM t Allan Collns Marlyn Jager Adams Rchard W. Pew 1 August 1976 Sponsored by Offce of Naval Research and the Advanced Research Projects Agency Approved for publc release; dstrbuton unlmted, Reproducton n whole or n part s permtted for any purpose of the Unted States Government. BOSTON WASHNGTON CHCAGO HOUSTON LOS ANGELES OXNARD

2 M >.v,-., ;'..:.\:.. The Effectveness of an nteractve Map Dsplay n Tutorng Geography ft;! Allan Collns Marlyn Jager Adams Rchard W. Pew Bolt Beranek and Newman nc. Cambrdge, Massachusetts Contract No. N C-0083, effectve September 15, Expraton Date, September 30, 1976 Total Amount of Contract - $187,000 Prncpal nvestgator, Allan M. Collns (617) Sponsored by: Offce of Naval Research Contract Authorty No. NR Scentfc Offcers; Dr. Marshall Farr and Dr. Joseph Young and OJ4# Advanced Research Projects Agency ARPA Order No. 2284, Amendment 5 Program Code No E NHR The vews and conclusons contaned n ths document are those of the authors and should not be nterpreted as necessarly representng the offcal polces, ether expressed or mpled, of the Advanced Research Projects Agency, the Offce of Naval Research, or the U S Government. Approved for publc release; dstrbuton unlmted. Reproducton m whole or n part s permtted for any purpose of the Unted States Government. S

3 ss? o,^. ^;!^SÖ^^a^äÄ-'^/S,^,o MnMMRsnn SLCURT Y CU ASSH K" AT ON OF TMT. P A.,'. E 'll A d /)(Ul l.lllrrrd), H rom NUMB L R REPORT DOCUMENTATON PAGE Techncal Report No. 2 / ^»(rtrffcr"*""'''"'*' 2. GOVT ACCESSON NO.-mrafrwKW».,»»»"»»«BsaBM^^j, JBE EFFECTVENESS OF AN NTERACTVE J1AP "TDSPEAY N.TUTORNG GEOGAPHYJ ^ 7. *UTHOR(Sl wwwmawwg'gwjgjjwj"^^'**^ HKA) N.STUCTONs ; ()HK COM'.-. M. -1 MM BBN Report No S. CONTRACT OR GRANT NUMBERS) -.. Al an/coll ^ ns, Marlyn Jager/Adam^, :/* Rchard/Pew l. PERFORMNG ORGANZATON NAME AND ADDRESS t. Bolt Beranek and Newman nc. / 50 Koulton St., Cambrdge, Ma CONTROLLNG OFFCE NAME AND ADDRESS Personnel and Tranng Research Programs Offce of Naval Research (Code 458) Arlngton, " xnqton, Va MV^röNrrffrt-wo-A&ENCXNAME.fc^AODFess^/.d/Me''/fO'n ControUngOffce) MttMrnanM^ag.««. N C PROGRAM ELEMENT PROJECT TASK AREA & WORK UNT NUMBERS 61153N RR NR ^> s / 15. SECURTY CLASS, (of thn refptrtj f Unclassfed DECL ASSFC ATON/ DOWNGRADNG SCHEDULE 16. DSTR JTON ST ATEMENT ^o/ l/lls /?epor(>.^ /-:. \7,Q\lt\*U T rto^ sj,*xzal*&'^^'k^a^umt^^ f rom l e P on ) 7 ^*fc(3»(vj«ö= ^^«**«maw.,vwjäjk^.^»,; W^MBltat Ul 'V ft *h y/ <? wt*w. JKCT «ml"^' 7 MOWMMMRS WHrwwwaMWryw To^b^^subm^ted^'tö^trüurttal^of^"^^ Psychology s f *4a' s ] S. KEY WORDS (Conthue on reverse äe/neeesary and fentfy by block nu Educaton, Tutorng, Computer-asssted nstructon. Learnng, Teachng, Educatonal Psychology, Evaluaton. 20. ABSTRACT ^Contnue on reverse sde f necessary and dentfy by block number) *"The purpose of ths study was to evaluate the teachng effectveness of dfferent aspects of the SCHOLAR CA system. The experment compared how well students learn usng SCHOLAR wth (a) the nteractve map dsplay of Map-SCHOLAR (b) a statc labeled map, and (c) an unlabeled map. The results of the experment showed that the students learned sgnfcantly more wth the nteractve map dsplay, than wth ether the labeled - DD t j 73 14^3 EDTON OF 1 NOV 65 S OBSOLETE -V.; /*)// /- / " ^ ') -_- ' V-.- r SECURTY CLASSFCATON OFTHSPAGEfBAen Dala - nfrt-jl

4 f. URTY CL Ar.Sl'lC Al ON OF THS PAGE (Sh-u JhaTnlrrr) vz: (20) 's -a..,js- lü.cjl... ^-7v\_cj(_-C-j map,^ A new method called backtrace analyss was used to assess the effectveness of specfc aspects of the tutorng strategy and the map system used n the experment, /o~y -,r.ft, ;.- -«ss "::. d SECURTY CLASSFCATON OF THS PAGE(9ken Data Entered)..:; ' ' *::: ^... n a,«.., «wn.,.tu «j.n.. awg

5 1 1...!-,.^ ^ ^ '':L..;.:-....,: :.,.. :,, ^., : ',:',:j../.::,^ ',- «/ 'k:r,,p^-^-' &. '','? :.1,.^..,:\:.UJ^^.:::^ r-.>-\-,-:/,:: :;., :.:: ::. - Lh-^^^^: ^--V^v^.l :j:n:f*-u '." ' - ' 1 The Effectveness of an nteractve Map Dsplay n Tutorng Geography Allan Collns Marlyn Jager Adams Rchard W. Pew Bolt Beranek and Newman nc. 50 Moulton Street Cambrdge, Mass Ths research was sponsored by the Personnel and Tranng Research Programs, Psychologcal Scences Dvson, Offce of Naval Research, under Contract No. N C-0083, Contract Authorty dentfcaton Number, NR We would lke to thank Nelleke Aello, Susan M. Graesser and Barbara N. Freeman for programmng the system and carryng out the experments and data analyses descrbed. U:

6 *: r ;" ' 8Ü l ABSTRACT The purpose of ths study was to evaluate the teachng effectveness of dfferent aspects of the SCHOLAR CA system. The experment compared how well students learn usng SCHOLAR wth (a) the nteractve map dsplay of Map-SCHOLAR (b) a statc labeled map, and (c) an unlabeled map. The results of the experment showed that the students learned sgnfcantly more wth the nteractve map dsplay, than wth ether the labeled map or the unlabeled map. A new method called backtrace analyss was used to assess the effectveness of specfc aspects of the tutorng strategy vv S and the map system used n the experment. : ;; S-: '' :!.1^^..;r.'-:.^;>^:,'-,^::^-:;v.v.v-._.. > -^ _..._...,..._._.._.._ _._.... '..._.._._ _. _ -.-..^.-^ w.r. y.':^ ' _^^ '

7 3 : '.^ ' ';,:. ;/, :V^. :.^-v. ^ ;-,.- 0.:..,.;,..;.. ;.-.- :: <:-';; :.'' ". >jv -V:': ^ J'r.M'-'-.X-^-'^:.-'vL^.;- NTRODUCTON n developng SCHOLAR, Carbonell (Carbonell, 1970; Carbonell and Collns, 1973) took a frst step toward a computer asssted nstructon (CA) system that s capable of conductng general tutoral dalogues wth students. n SCHOLAR, knowledge s not stored as text, but n a precselystructured semantc network of nterrelated facts and concepts (Qullan, 1968; Collns and Qullan, 1972a). Every concept used to descrbe a gven concept can tself be descrbed elsewhere n the network. Thus, n a nontrval sense, the program can understand the concepts t uses. SCHOLAR also has dfferent subroutnes that use the structure of the network to formulate questons for the student, evaluate hs answers, answer hs questons, make nferences and computatons, select new topcs for dscusson, etc. The attempt r to structure nformaton lke a human knowledge, so that the program can use ts knowledge as flexbly as a human tutor does. Because of ts nherent flexblty, the system can be extended n a varety of ways. For example, SCHOLAR has been modfed to teach dfferent knds of knowledge (Collns and Grgnett, 1975) and to use dfferent teachng strateges (Collns, Warnock, and Passafume, 1975). The educatonal queston s whch of the possble ways of varyng SCHOLAR ncrease ts teachng effectveness. Ths

8 9 1..:.-.;:.',....--v/.;.:.'':... ;,,. '...V.1... queston s beng nvestgated by runnng systematc expermental evaluatons of dfferent system confguratons and dfferent teachng strateges (Collns, '\97 \). The experment reported here was desgned to test the utlty of the map capablty recently added to the orgnal SCHOLAR program for teachng geography. The Map-SCHOLAR system was developed to ntegrate the tutorng of graphc nformaton wth verbal nformaton. n vew of the evdence that pctoral nformaton may be remembered more easly than verbal nformaton, (c.f. Bower, 1972; Pavo, 1971) we expected ths capablty to ncrease SCHOLAR'S teachng effectveness. Map-SCHOLAR can dscuss wth the student dfferent maps that change dynamcally accordng to the context of the s dscusson. To do ths, a graphc structure was created whch parallels the structure n the semantc network. The elements n the map dsplay can be referred to ether by ther name, or by pontng to them, or both. Map-SCHOLAR both asks and answers map-related questons and provdes relevant map nformaton when the student makes a mstake. n short, Map-SCHOLAR has all the capabltes of the orgnal SCHOLAR wth the addton of the map capabltes. Fgure 1 llustrates some of the varety of nteractons possble wth Map-SCHOLAR. The frst panel llustrates how Map-SCHOLAR asks map-related questons. """^^' ^^vwummma^mma^tmsvkmw^-lst^st

9 SX; ;',\..;.o'v';-.,,.'.' '.:. /'^^^^.^.^v'-.-/..'r'':^.v' ^U:^k.l;;^.'!^.,-^':.' :..;.-.;> -),! ' -.f'-'.-^ -v.rv-t' NOW LET'S SEE WHAT YOU REMEMBER ABOUT THE CTES OF BRAZL. PLEASE NAME THE PRNCPAL CTES OF BRAZL NOW BLNKNG ON THE DSPLAY SCREEN. *MANAOS SAO PAULO AND LMA* HANAUS AND SAO PAULO ARE CORRECT. LMA S N PERU, NOT N BRAZL. THE PRNCPAL CTES OF BRAZL ARE BLNKNG ON THE SCREEN. RO DE JANERO S N SOUTHEASTERN BRAZL. T S ON GUANABARA BAY AND THE ATLANTC, BRASLA S N CENTRAL BRAZL. T S FAR FROM THE OCEAN. RECFE 0 DE JANERO ORTO ALECRE Fg. 1 nteractons wth Map-SCHOLAR '& V: ' ' _""*

10 .. PLEASE r TO THE FOLLOWNG CTES OF THE AMAZON N THE ülder M WHCH THEY APPEAR BELOW: MANAOS, BELEM, AND QUTOS DON'T FORGET TO LET ME KNOW WHEN YOU ARE FNSHED, PLEASE REMEMBER TO TYPE.AN ASTERSK * FOLLOWED BY A CARRAGE RETURN. ** MANAOS S CORRECT. YOU DD NOT PONT AT BELEM AND QUTOS CORRECTLY, THE CTES OF THE AMAZON ARE BLNKNG ON THE SCREEN. ^ a : /,; ' ; ' m 1 Fg. 1 nteractons wth Map-SCHOLAR (cont) ^Lj.--k y^s.'jüu*

11 mmmm. J:WJ.J *. 1! 1.' 4 LET'S BEGN BY TALKNG ABOUT THE TOPOGRAPHY OF VENEZUELA. THE PRNCPAL LAKE OF VENEZUELA S BLNKNG, PLEASE NAME T. * DON'T KNOW* THE PRNCPAL LAKE OF VENEZUELA S BLNKNG ON THE SCREEN. E LAKE MARACABO S N NORTHWESTERN VENEZUELA. THE PRNCPAL MNERAL S OL. *S THS COUNTRY COLOMBA?* YES, THAT'S COLOMBA *WHERE S THE CAPTAL OF VENEZUELA?* CARACAS S BLNKNG. m Fg. 1 nteractons wth Map-SCHOLAR (cont) n HP _-, MP^^SSäSfW.«nllMMHHmWW m «-»«unllm llmllamlmmwmmmmlllmw^^

12 E evaluates the student's answers, and corrects hs mstakes. Frst, the dots ndcatng the locaton of the ctes appear unlabelec on the map of Brazl and start blnkng. Then SCHOLAR asks the student to name the blnkng ctes. When B he responds, t tells hm whch answers were correct and prnts ther names on the map. Because he erroneously gave Lma as an answer, Map-3CH0LAR ponted out that Lma s n Peru and showed where Lma s on the map. Thus, Map-Scholar gves both verbal and vsual feedback to help the student learn the materal. The second panel shows how context affects the map dsplay. The topc s the Amazon, whch appears n ths t context on the map of Brazl, the overall topc. The queston asks the student to pont at the ctes along the Amazon n the order named. The dots for the three ctes frst appear unlabeled, and only after the student has answered do the labels appear. Unlke the frst map, ths map shows only the ctes on the Amazon and ther relaton to the Amazon. There s also expanson of detal so that a cty lke Belem, whch was not mportant enough to be ncluded among the ctes of Brazl, s ncluded wth the subtopc of the Amazon. Because the map changes dynamcally as the context changes and as the student nteracts wth the system, the student's attenton s focused on the relevant nformaton, and questons can be posed n a vsual form not E possble wth a statc map dsplay. 7 ^ ' ^ V'v"^*,. ;V\V:. ;-:rv;: "'TV..;-"' " '' ^ ; >:.-V-..-~..../;- -. :..* -... ^.^v / JW^:--..,'-^ ^-><.. ->,-...:.-. : j-r:....^,-^^..- BMW«KS6at^.--3S.1)^l:C(.ü'W^

13 Lü-l^ "' Ja '-- The thrd panel shows the system's potental for tutoral nteracton. When the student ddn't know about Lake Maracabo, Map-SCHOLAR showed t on the screen and added the related verbal nformaton about the ol there. Ths example also llustrates some of the ways n whch the student can ask Map-SCHOLAR to clarfy or amplfy the nformaton gven (Collns and Warnock, 1974). For the frst queston the student both ponted at and named Colomba to ask f t s the country near Lake Maracabo. For the second queston, the student verbally asked where the captal of Venezuela s, perhaps to fnd out how far away t s. SCHOLAR fgured cut semantcally that the captal 3 Caracas, and then vsually showed where Caracas s by blnkng t (t s the double square). These examples llustrate some of the power for tutoral nteracton that can be obtaned by a close ntegraton between semantc and vsual knowledge. n order to test the utlty of the map system for teachng, we conducted an experment n whch each student learned about a dfferent country under one of three condtons: one condton used SCHOLAR on the map system; the second condton used SCHOLAR on a non-graphc termnal, but the student could look at a labeled map of the country; the thrd condton was lke the second, except that the student was gven an unlabeled map Students' learnng for each of the three knds of tranng sessons was measured by m 8»^«SWH^«^^

14 comparng ther scores on a pre-test to those on a post-test gven three days after the last tranng sesson. A second goal of th; experment was to nvestgate how specfc aspects of the tutoral dalogue affect students' learnng. To study ths queston, we developed a technque called backtrace analyss. The technque nvolves markng each pece of nformaton that s dscussed accordng to the knd of exchange nvolved (e.g., a queston requrng a pontng response vs. a namng response). By comparng ths data to the student's answers on the post-test, t s possble to dentfy the knds of tutoral nteractons that most strongly nfluence the student's learnng. METHOD v.j : Subjects. The ntal group of subjects ncluded nne hgh school students. The study was replcated wth nne unversty ^udents. All subjects were volunteers and were pad for ther servces. Desgn. There were three expermental condtons: a Map-SCHOLAR condton, a Labeled Map condton, and an Unlabeled Map condton. The Map-SCHOLAR condton was run on an mlac graphc termnal wth the screen dvded between maps and verbal communcatons as shown n Fgure 1. The student could nput questons and answers by a keyboard and an electronc ponter (a "mouse"). The Labeled and "':mm ' " ^'' ""ywgfflpwwwajbm

15 -'..:: : V '. Unlabeled Map condtons were run on a keyboard termnal usng a non-graphc verson of SCHOLAR called Tutor-SCHOLAR (Collns, Warnock, and Passafume, 1975). The two versons of SCHOLAR were dentcal wth respect to both teachng strategy and nformaton n the data base, except that Map-SCHOLAR handled all locaton-related questons n terms of the map, whereas Tutor-SCHOLAR handled them verbally. n the Labeled Map condton, subjects were gven a paper map whch marked all the places (names and locatons) ncluded n the Map-SCHOLAR data base. n the Unlabeled Map condton, subjects were gven copes of the same maps, omttng the names of the places that were marked on the maps. For both of these condtons, students were nstructed not to mark on the maps. The pre-test, post-test, and the fnal questonnare were gven n paper and pencl format. Procedure. Each student partcpated n a prelmnary sesson, three tutoral sessons, and a post-test sesson. The frst purpose of the prelmnary sesson was to admnster the pre-test. The pre-test measured the student's pre-expermental knowledge about the nformaton to be tutored, and conssted of 20 basc questons about the geography of each of the three relevant countres: Argentna, Brazl, and Venezuela. A secondary purpose of the pre-test was to ascertan that no subject was nordnately famlar or unfamlar wth any one of these 10 S^SJ^SÄW^ÄKoss-wcea»! ^

16 j «L«m.Ku n..mw.»..ll..'.."gttr f three countres, snce such nequaltes n pror knowledge would confound measures of teachng effectveness. After havng completed the pre-test, the student was gven a bref, ntroductory lesson on a fourth country, Chle, usng Map-SCHOLAR. The; purpose of ths lesson was to famlarze the student wth the system and ts capabltes or, more specfcally, wth the knds of questons he would be asked, the knds of answers that were expected of hm, the knds of questons he could ask of SCHOLAR, the use of the keyboard and the ponter, and the methods by whch he could correct hs nput errors. t;: The tutoral phase of the experment conssted n three, two hour sessons, admnstered on consecutve days. Durng these sessons, each student learned about one country n the Map-SCHOLAR condton, one n the Labeled Map condton, and one n the Unlabeled Map condton. Each lesson lasted for one hour. After the student had receved one lesson on each of the three countres, the seres was repeated. The combnatons of countres and teachng modes were counterbalanced and ordered accordng to a 3 x 3, confounded, factoral desgn (Wner, 1971, p. 6^6). The fnal sesson was conducted three days after the last tutoral sesson. n ths sesson, the student took the post-test and completed a questonnare on those aspects of the lessons that he had found most and least helpful. n

17 _..._._. _._ :......,.,..._^..,-' The post-test was dvded nto three parts. The frst part conssted of 36 basc questons (ncludng the 20 that had been on the pre-test) about each of the three countres. For the second part of the post-test, the student was gven a map of each of the three countres and asked to label the geographcal features ndcated. The thrd part of the post-test conssted of 32 more dffcult questons about each of the countres. Backtrace Analyss. n order to assess the value of specfc aspects of the tutoral exchange, we developed the technque of backtrace analyss. Ths technque nvolves markng each entry n SCHOLAR'S database wth respect to the way n whch t s treated durng a gven tutoral sesson. Ths nformaton can subsequently be retreved, enablng us to evaluate the effectveness of SCHOLAR'S varous nteractve capabltes from the probabltes wth whch they result n correct answers on the post-test. More specfcally, each tem that was dscussed n a gven sesson was tagged wth nformaton concernng (1) the temporal order, (2) the context, and (3) the tranng event n whch t arose. For purposes of the backtrace analyss, the tranng events were classfed as follows: a) True-False Correct - SCHOLAR presents a true-false queston whch the student answers correctly. SCHOLAR tells the student he s correct and moves on to new 12

18 - -. >*^ M ll-ml ftmu. LL'g ;J1 m a! m 1 E nformaton. b) True-False Error - SCHOLAR presents a true-false queston and the student answers ncorrectly or pleads gnorance. SCHOLAR ponts out the correct answer and goes on. c) Name Correct - The student correctly names a geographcal feature(s) n response to SCHOLAR'S request. Each answer among a set of answers s tagged ndvdually. Ths category subsumes what and where questons as well as fll-n-the-blanks and namng requests by SCHOLAR. d) Name Error - The student ncorrectly names or fals to name a geographcal feature when questoned by SCHOLAR. e) SCHOLAR Error Correcton - f the student completes a fll-n queston erroneously, SCHOLAR nfers the bass of the student's error and then presents new nformaton to dstngush between the student's answer and the correct answer. f) SCHOLAR elaboraton - f the student msses a queston, SCHOLAR presents related nformaton at the same level of mportance (See Fg. 1). The related materal s tagged as an elaboraton. g) Student Queston - nformaton s ntroduced as the result of a queston the student asks of SCHOLAR. n addton to the above, there were several categores of 'tranng events whch occurred only n Map-SCHOLAR. SCHOLAR treated these events lke fll-ns, but they were 13 VfSjSröK rjtpftqrw*«*»" * "

19 dstnctvely marked for purposes of the backtrace analyss: h) Label - SCHOLAR asks the student to name those features of the map that are blnkng, ) Pont - SCHULAR asks the student to pont to the specfed geographcal features on the map. j) Label and Pont - SCHOLAR asks the student to name and pont to a specfed set of geographcal features. RESULTS AND DSCUSSON The pre-test scores were examned usng a 3 X 2 (Countres X Groups) repeated measures analyss of varance (Wner, 1971, p. 518). The only sgnfcant effect was due to groups, as the college students generally scored hgher than the hgh school students. The number of correctly answered questons, out of the possble 20 per pre-test, ranged from 1 to 11 (medan = 4.67) for the college students and from 0 to 5 (medan = 0.64) for the hgh school students. Nether the man effect of countres (F(2,32) = 2.62, p>0.05) nor the nteracton between countres and groups (F(2,32) = 1.36, p>0.05) approached sgnfcance. nasmuch as none of the subjects knew much about any of the countres n advance, the dl ce between pre- and post-test scores should prov le far estmate of SCHOLAR'S teachng effectveness. Moreover, snce the subjects' pror knowledge seemed to be evenly dstrbuted 14

20 across countres, the relatve teachng effectveness of the three condtons could be estmated through drect ;: comparsons of the correspondng pre-test/post-test dfference scores. Wl :. :::. The average ncrease n the number of correct responses from the pre-test to the post-test s shown n Fgure 2 for each teachng mode. These dfference scores were analyzed accordng to a 3X3 (teachng modes X countres) confounded factoral desgn. Whereas nether the effect of countres (F(2,28)<1.0) nor the nteracton between countres and teachng condtons (F(4,28) = 2.08, p>0.05) was sgnfcant, the effect of tranng condton was strongly sgnfcant (F(2,28) = 6.05, p<0.01). Accordng to a Newman-Keuls test (p<0.01), the Map-SCHOLAR condton resulted n sgnfcantly hgher post-test scores than the Labeled Map condton whch, n turn, resulted n sgnfcantly hgher scores than the Unlabeled Map condton, Separate analyses of the three parts of the post-test ndcated that much of the effect of teachng modes occurred n the part of the test consstng of map labelng questons (F(2,28) = 14.09, p<0.001). However, a pronounced effect of teachng mode was also obtaned for the easer, non-map questons n the frst part of the post-test (F(2,28) = 5.85, p<0.01). Although the 15 MM^MVBMNlHMBBamaNHnBBBB ^^^^^^...:...:...' ^.;.,,%';-:^v:,--;.;.-:...; ;. t j--^-^a..;-.

21 fe - ;,: \ : <; -.: : : : :...".-....,. : -;;S ' ;-.: 'v-->,.".v-\.j..^ -.';: :.....,. -:.':-., _,',';. J-S CM LL) CD o ~ o < LL. < ^ CO 5 < Q Q 2 LL! O LL) - o h- LU Z X 3 h- JJ 11 o z < H D Q z o Q O Ü ro ^ m CVJ o CV n n :> ls31-ls0d QNV ls31-3dd N33M13a 3d03S 33N3a3JJa 39Vd3AV 16 ^--v^.'.v^-;:. v^^-.,r.;'-\^^^:-;^^^-!.---^^-v?;r-^/^"^r-;-^:\v-y-^.7(.^^

22 l/^^-^-^-..^.;:r';j.v-:-".v '...-; -_^.1 v ^ '. ' ' ''" ".'' scores on the more dffcult questons n the thrd part of the post-test were too nosy to yeld any sgnfcant effects or nteractons under analyss, the same trend was apparent. n short, post-test scores were consstently hghest n the Map-SCHOLAR condton and lowest n the Unlabeled Map condton. These results ndcate that the map system not only helped students learn the nformaton necessary to answer the map questons n Part 2 of the post-test, but also to answer the verbal questons n Parts 1 and 3 of the post-test. An mportant queston s whether the beneft of the map system extended only to verbal nformaton that was explctly about locatons, or whether t also extended to non-locaton nformaton, such as the clmate or terran of a place. Clearly, one would expect the map?;' ' system to help students learn locaton nformaton better, but there are two reasons why the map system mght help students learn non-locaton nformaton better as well. Frst, f map nformaton showng where a place lke Manaos s located helps the student remember Manaos better, then non-map facts about Manaos, such as ts clmate, may be remembered better. Ths s because the best way to learn somethng s to relate t to nformaton already known (Collns and Qullan, 1972b; Norman, 1973). Second, f a student sees that Manaos s on the Amazon, then Manaos' clmate can be related to any 17

23 '. --. ; '>' r : )^-u''?-.', pror knowledge about the clmate of the Amazon (e.g. that the Amazon flows through jungle). Thus, even non-map nformaton may be better remembered n a vsual context. Ths dea was tested wth backtrace analyss by separatng the questons durng tranng nto map questons and non-map questons, dependng on whether the questons were posed vsually by the map system. Then the percentage correct on the post-test for the two types of presentaton durng tranng were plotted (see Fgure 3). For map questons, as expected, students learned sgnfcantly more wth Map-SCHOLAR than wth ether the labeled or unlabeled maps. But for non-map nformaton there were no sgnfcant dfferences, and students even dd slghtly better n the Unlabeled Map condton. Thus, these data suggest that the major beneft of the map system s n learnng nformaton about specfc locatons. Backtrace analyss was also used to nvestgate the effectveness of repeatng questons, dependng on whether the student answers correctly or ncorrectly. Fgure 4 shows the percentage of correct responses to each tem on the post-test as a functon of how frequently the students were correct or wrong on that tem durng tranng. The ncreases n the curves show 18 '. ;.;:.,. ' -,-.,,.>. : -. ^.- ^"-^

24 mmam um «mmnnm.nmüm.wmnnjum' l CO o: CJ3 g H Q z o ÜJ < O UJ < cc O c^ L_ Z o: o UJ r: ^^ H UJ c^ ^ UJ o CD Z z < 01 \- Ü z or a z o o z o CO ^ z: a. < S31-S0d NO 103dd03 ln33d3d lo: 19

25 o 2 O 0 - ^ h- tfc o 2: -z. <r r) rr u_ h- < 0 </) 2 < cc: 1-3 ÜJ rr H 0 f~ or C/) LL O Lü * 0. -z. - LÜ 1 " S ^ " HO U- 2H O O h O a: 0 cr 0 o: 0 j m "Z. ÜJ _) O -^ CC Lü a. 1S31 SOd NO ld3üyoo NBOüSd

26 : ;; : ''-:...,....; '.^^A^'.jv.hL.,j/.s.:;.-.^.^..:>j;;,' that the more frequently a student answered any tem correctly, ehe more lkely he was to recall t on the post-test. The separaton of the curves for dfferent numbers of errors shows that the more frequently an tem was mssed, the less lkely t was to be recalled on the post-test. Ths smply reflects the fact that the tems that were more dffcult to learn were lkely to be mssed more frequently. The concave shape of the curves ndcates that the repeated presentaton of a correct tem has a decreasng effectveness. The mplcaton s that, as far as possble, tranng tme should be allocated to those tems that the student has correctly answered least often. When students mssed tems n answerng a queston, SCHOLAR provded addtonal elaboraton about some of the tems mssed. For example, n Fgure 1 when the student dd not know Lake Maracabo, SCHOLAR mentoned the ol n OTT! H ll m ü 81 $' :":;J there as an elaboraton about Lake Maracabo. The backtrace analyss showed that percent correct on the post-test ncreased from 3^% when there was no elaboraton of an tem durng tranng to 47$ when there was one elaboraton. Ths ncrease s sgnfcant (t = 4.01,p<0.01), ndcatng that elaboraton does help students to learn the materal better. After one elaboraton, the percent correct stablzes, ndcatng that further elaboratons are of lttle beneft. 21 f

27 We used a varaton of backtrace analyss to determne whch knds of map questons are most effectve for learnng. n the map system there were three dfferent knds of map questons that mght be asked: (1) pontng questons, where SCHOLAR mentoned one or more places and asked the student to pont at them (2) namng questons, where SCHOLAR blnked one or more places and asked the student to name them, and (3) pontng and namng questons, where SCHOLAR asked the student to name a set of places, such as the rvers n Brazl, and pont to them n the order named. Fgure 5 shows the percent correct on the second occurrence of a map queston about any tem as a functon of the type of queston that was asked on the frst occurrence of that 'em. There were not enough data for namng questons, so they are not shown. The column totals ndcate that students dd better on pontng questons than on pontng and namng questons, as would be expected because pontng questons are easer. However, the row totals show that students dd better on the second queston f the frst queston had requred both pontng and namng than f t had requred only pontng ofl) = 4.75,p<0.05). Evdently, students learn more from pontng to and namng a locaton than from Just pontng to t. 22

28 ^^^rt Vfl^^B? wy«-^ - Vlfv:,.^ ^-r.t-f::'. q 6 v a. m SB y; f CO Q. >- H CD 2 < QjCD a: ^ Lü or to Q CO o o h-hoco Lü Lü LL 3 U. O LU üü X ÜL < UJ U z UJ a: 80 H- O 2 Z UJ O O O CC LU UJ CO o z o a. O $5 ro S5 05 o z 1- z o a. n S5 0^ 5 o a o z z z <p <? z o a. 0 s ro n O V a. 4 L, o UJ z Z QJ OQ:»-^ UJO ^^ OO OlO a: UJ Q.U. > KZ O 23 ^v-t-q :^ rx-?r:;z \r:.vc^>r.xaamsa!^^ ': - :. :.J-C ::% ': ^

29 CONCLUSON The experment showed that students learned sgnfcantly more wth the nteractve map dsplay than wth ether a statc labeled or unlabeled map. The advantage of Map-SCHOLAR cannot be attrbuted solely to the ablty of the student to locate places spatally, snce the Labeled Map condton allowed the student to dentfy places just as effectvely. The advantage of Map-SCHOLAR also cannot be attrbuted to novelty or some other generalzed facltaton effect, because, as backtrace analyss showed, the effect was specfc to locaton nformaton and dd not carry over to non-locaton nformaton. The advantage therefore must have been due manly to the dynamc aspects of Map-SCHOLAR and ts ablty to focus the student's attenton on the relevant map nformaton. The experment also demonstrated the usefulness of the backtrace analyss technque for evaluatng CA systems. Backtrace analyss s not dependent on the type of nformaton beng taught, and s thus transferable to CA systems other than SCHOLAR. Of course, the specfc tags used to mark the data would change, dependng on the dfferent teachng strateges and tranng events that are beng evaluated. The ablty to perform fne-gran analyses of the effectveness of dfferent teachng 24

30 f...' "..-., '-^.'. ':'.-. " - ;- -. % '- ' : '. '.' * -r. : ' - -, ^-.-. " ;';>. " -.'.,..,,_...,,..., strateges s a valuable tool for future educatonal research. -f; P t }\ ^! p ^. ra»a 1 «fn 11 Eäfe 1^ 25

31 m m ^W5^!55W,»^a^»W-?«^^K»^^^ '-' ^ K *,1^^TW^?,ä y3^^p^^ ppp^h3^p!3 ÜÜ,..,.y;:.. /.-v'.... : : -.:,.,^,,;. v..v:.: :..-!v :v.',-.ä*,;::v;,. AMmMMMsä REFERENCBS Bower, G.H. Mental magery and assocatve learnng. n L.W. Gregg (Ed.), Cognton n Learnng: and Memory. New York: Wley, Carbonell, J.R. A n CA: An artfcal ntellgence approach to computer-aded nstructon. EEE Transactons on Man-Machne Systems. 1970, MMS-11, Carbonell, J.R. and Collns, A.M. Natural semantcs n artfcal ntellgence. n Proceedngs, of the Thrd nternatonal Jont Conference on Artfcal ntellgence, 1973, 3^-351. Collns, A. and Grgnett, M.C. ntellgent CA. BBN Report Mo. 3181, To be submtted to nstructonal Scence. Collns, A.M. Comparson of two teachng strateges n computer-asssted nstructon. BBN Report No. 2885, Submtted to Contemporary Educatonal Ps.ychol og^. Collns, A.M., and Qullan, M.R. How to make a language user. n E. Tulvng and W. Donaldson (Eds.), 26

32 >:. «, ',-....w.... '..,,^ Organzaton of Memory. New York: Academc Press, 1972a. Collns, A.M. and Qullan, R.M. Experments on semantc memory and language comprehenson. n L.W. M m Gregg (Ed.) Cognton n learnng and memory. New York: Wley, 1972b. m MM Collns, A.M. and Warnock, E.H. Semantc networks, BBN Report No. 2833, ml r.» Collns, A.M., Warnock, E.H., and Passafume, J.J. Analyss and synthess of tutoral dalogues. n G.H. Bower (Ed.) The psychology of learnng and motvaton, Vol. 9 New York: Academc Press, Norman, D.A. Memory, knowledge, and the answerng of questons. n R.L. Solso (Ed.), Contemporary ssues n cogntve psychology: The Loyola symposum. New York: Halsted Press, m $....jj 1 g m E :./: :' :[-::^^V$^^ Pavo, A. magery and verbal processes. New York: Holt, Rnehart, and Wnston, Qullan, M.R. Semantc Memory. n M. Mnskey (Ed.), Semantc nformaton processng. Cambrdge, Mass.: MT Press,

33 'äkwwkwwsmbw«^8^ Wmmä::^ ;.:;:,. :>.. S':rf, ä : :;.;w,&;..._ :, :,...;..V,\.,; : ;. ;,,,,,.,. :._,,,,,, r**~4 Wner, B.J. Statstcal prncples n ex^ermental desgn. New York: McGraw Hll, 197t, 28

34 '- %, " :. ;... ;- :. -v':-:.:' -.;... ^ -'.:.:^0,.-'.:."--^-^-.-^; : --:- ; "..,' 1 W-:'-, ; '/--,'v.._,-, c,.-;-'^.%;;:'.:;;.' c^vv'-vu.- 1 '- 1 ;'.^.:-^^^ ^'^ ^^^^j:-^.^;vvv::/^^^o> ^^ ;rvvl ^ : ^ ; -J;^ ;:v; -v^/^/:^^;^v ;;'-1 ^: v.",. -.;.' ; ;; '^ 1 ^ v v ^:^^ ^;'v.-.^; ^-^ H DSTRBUTON LST Dr. Marshall J. Farr, Drector Personnel & Tranng Research Programs Offce of Naval Research Arlngton, VA 22217? as Sa TO mm Drector ONR Branch Offce 495 Summer Street Boston, MA ATTN: Dr. James Lester Drector ONR Branch Offce 1030 East Green Street Pasadena, CA ATTN: E.E. Gloye Drector ONR Branch Offce 536 South Clark Street Chcago, L ATTN: Dr. Charles Davs Dr. M.A. Bertn, Scentfc Drector Offce of Naval Research Scentfc Lason Group/Tokyo Amercan Embassy AP0 San Francsco Offce of Naval Research Code 200 Arlngton, VA H W 1 Assstant Deputy Chef of Naval Personnel for Retenton Analyss and Coordnaton (Pers 12) Room 2403, Arlngton Annex Washngton, DC Dr. Lee Mller Naval Ar Systems Command AR-413E Washngton, DC Commandng Offcer Naval Health Research Center San Dego, CA ATTN: Lbrary AS v.5>:?üfts-bny'/p;f-"'f.'^''.".':r"v..--.::~:.t.'!^'k., swcä:

35 ük :, :^ :. : '.< ::. -...^.,.., ' ' '- : ' ' ' -.^ ' ^ * :' ' '' '!S!p«$!ep8!S!^^yt : v''^ ^ : ". ' Page Drector, Navy Occupatonal Task Analyss Program (NOTAP) Navy Personnel Program Support Actvty Buldng 1304, Bolng AFB Washngton, DC Offce of Cvlan Manpower Management Code 263 Washngton, DC Chef of Naval Reserve Code 3055 New Orleans, LA Chef of Naval Operatons OP-987P7 Washngton, DC ATTN: CAPT H.J.M. Connery Drector Tranng Analyss & Evaluaton Group Code N-OOt Department of the Navy Orlando, FL ATTN: Dr. Alfred F. Smode LCDR C.F. Logan, USN F-14 Management System COMFTAEWWNGPAC NAS Mramar, CA Navy Personnel Research and Development Center Code 01 San Dego, CA Navy Personnel Research and Development Center Code 02 San Dego, CA ATTN: A.A. Sjoholm Navy Personnel Research and Development Center Code 306 San Dego, CA ATTN: Dr. J.H. Stenemann

36 ....: :...,.. ;.... V.,:. - T^,.'-,. v.>:,y,.,:' :.....,;,,....,.: s,..v-,; V..V''.r/'/.. ^..:',,^.v..;v;:v,^..:->;/...'v.^: ;.^,.:,^.-..V,. :;.^^: ;.v,^.. ;. :.. ' :.. -^.-y ^. - - '""'" '- --'' - Navy Personnel Research and Development Center San Dego, CA ATTN: Lbrary Navy Personnel Research and Development Center Code 9041 San Dego, CA ATTN: Dr. J.D. Fletcher D.M. Gragg, CAPT, MC, USN Head, Educatonal Programs Development Department Naval Health Scences Educaton and Tranng Command Bethesda, MD STOAC Battelle Memoral nsttute 505 Kng Avenue Columbus, OH Attenton: H8-1 Code Drector Naval Research Laboratory Code 2627 Washngton, DC Pane m Defense Documentaton Center Cameron Staton, Buldng Duke Street Alexandra, VA Charman Behavoral Scence Department Naval Command and Management Dvson U.S. Naval Academy Luce Hall Annapols, MD m Chef of Naval Techncal Tranng Naval Ar Staton Memphs (75) Mllngton, TN ATTN: Dr. Norman J. Kerr Ü w,. summ

37 : fewaß&fc,,,._..,.,,...,.,,,,..,..,,..,..._,.,,.,,^:,,.,,:...,,:,..,, Pape 4 Chef of Naval Tranng Naval Ar Staton Pensacola, FL ATTN: CAPT Bruce Stone, USN TACTEC Battelle Memoral nsttute 505 Kng Avenue Columbus, OH Attenton H8-1 Code Defense Contract Admnstraton Servces Regon 666 Summer Street Boston, MA Attenton: DODAAD Code 52202A Command Offcer U.S. Naval Amphbous School Coronado, CA Dr. James J. Regan Techncal Drector Navy Personnel Research and Development Center San Dego, CA Chef Bureau of Medcne and Surgery Code 413 Washngton, DC Mr. Arnold Rubnsten Naval Materal Command (0344) Room 1044, Crystal Plaza 5 Washngton, DC Superntendent Naval Postgraduate School Monterey, CA ATTN: Lbrary (Code 2124) Chef of Naval Tranng Support Code N-21, Buldng 45 Naval Ar Staton Pensacola, FL * ÄJ^P^^'-^^^V^K^MMaBaÄ HLjHWnmM! «ss&tttwvwktmwww^

38 :'-\:- ^y^^j-t^ --::^A^^^:~:\:' y. ::Ljt- : -:-b '- ;'. ;:- '.-!.. ;! ^^- VJ^'^VV.-^'.-^-':^ y Dr. H. Wa ace Snako c/o Offce of Naval Research (Code 450) Psychologcal Scences Dvson Arlngton, VA Armed Forces Staff College Norfolk, VA dszu ATTN: Lbrary Mr. George N, Grane Naval Shp Systems Command SEA 047C12 Department of the Navy Washngton, DC Commandng Offcer Servce School Command U.S. Naval Tranng Center San Dego, CA ATTN: Code 3030 Dr. Wllam L. Maloy Prncpal Cvlan Advsor for Educaton and Tranng Naval Tranng Command, Code 00A Pensacola, FL HQ USAREUR & 7th Army 0DCS0PS USAREUR Drector of GED APO New York 09^03 AR Feld Unt - Leavenworth Post Offce Box 3122 Fort Leavenworth, KS Dr. Mlton S. Katz, Chef ndvdual Tranng and Performance Evaluaton U.S. Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA Techncal Drector U.S. Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA «^ ^^^v^w'w.^;; ^ Page U:. rj5jwy- o -'* ff, ' ÄM,M ' *" w '"'

39 P^Spy^SSa^f^,.,;; j.^.vr..,:^::;:^^--;^.«.*:'^^,-^-;:^-,^ > V ;^"-; ; '-^,'; j:,-;-r,;» ': ;-:''.' -o.wl Pape 6 Dr. Frank Harrs U.S, Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA Dr. Joseph Ward U.S. Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA Mr. James Baker U.S. Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA Commandant Unted States Array nfantry School ATTN: ATSH - DET Fort Bennng, GA U.S. Army Research nsttute Commonwealth Buldng Room Wlson Boulevard Arlngton, VA ATTN: Dr. R. Dusek Dr. Stanley L. Cohen Work Unt Area Leader Organzatonal Development Work Unt Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA Dr. Leon H. Nawrock U.S. Array Research nsttute Rosslyn Commonwealth Buldng 1300 Wlson Boulevard Arlngton, VA Dr. Martn Rockway Techncal Tranng Dvson Lowry Ar Force Base Denver, CO ; : )

40 '. :',. E E - Research Branch AF/DPMYAR Randolph AFB, TX AFHRL/DOJN Stop 63 Lackland AFB, TX ,...,..,,.,? «g p Pape 7 m ä m,$t; 1. 4 Dr. Alfred R. Fregly Ar Force Offce of Scentfc Research/PM Bolng Ar Force Base Washngton, DC Dapt. Jack Thorpe, USAF Flyng Tranng Dvson AFHRL/FT Wllams AFB, AZ AFHRL/PED Stop 63 Lackand AFB, TX nstructonal Technology Branch AF Human Resources Laboratory Lowry AFB, CO AFHRL/AS (Dr. O.A. Eckstrand Wrght-Patterson Ar Force Base OH AFHRL (AST/Dr, Ross L. Morgan Wrght Patterson Ar Force Base) OH Headquarters Electronc Systems Dvson ATTN: Dr. Sylva R. Mayer/MCT LG Hanscom Feld Bedford, MA Drector, Offce of Manpower Utlzaton Headquarters, Marne Corps (Code MPU) MCB (Buldng 2009) Quatco, VA r^ Chef, Academc Department Educaton Center Marne Corps Development and Educaton Command Marne Corps Base Quatco, VA :^_,-^^;2^;_:^.v : :-.:.-..yr.:-:-y:h-^:.^:^ l^; JJ._... : : ^_ - ; ; '

41 Pape 8 Mr. E.A. Dover 2711 f-cuth Vetch Street Arlngton, VA Dr. A.K. Slafkosky Scentfc Advsor (Code RD-1) Commandant of the Marne Corps Washngton, DC Mr. Joseph J. Cowan, Chef Psychologcal Research Branch (P-1) U.S. Coast Guard Headquarters 400 Seventh Street, SW Washngton, DC Dr. John Ford, Jr. Navy Personnel Research and Development Center Code 304 San Dego, CA LCDR Charles Thesen, Jr., MSC, USN, 4024 Naval Ar Development Center Warmnster, PA Mltary Assstant for Human Resources Offce of the Secretary of Defense Room 3D129, Pentagon Washngton, DC 203 v " Advanced Research Projects Agency Admnstratve Servces 1400 Wlson Blvd Arlngton, VA ATTN: Ardella Holloway Dr. Harold F. O'Nel, Jr. Advanced Research Projects Agency Human Resources Research Offce 1400 Wlson Boulevard Arlngton, VA Dr. Robert Young Advanced Research Projects Agency Human Resources Research Offce 1400 Wlson Blvd Arlngton, VA 22209

42 mmwmmmww?w ^^ ' "' " '< - ^' ^[ [^^ : :-'--- jr: -''' ', -: -'-.-:-..,..-...,. -:.-:...., Ov [3- Pape 9 v;- :; ä : : Ü 1 f 1 Dr. Wllam Gorham, Drector Personnel Research and Development Center U.S. Cvl Servce Commsson 1900 E Street, N.W. Washngton, DC 20^15 Dr. Marshall S. Smth Assstant Actng drector Program on Essental Sklls Natonal nsttute of Educaton Brown Buldng, Room th and M Streets, N.W. Washngton, DC Dr. Carl Frederksen Learnng Dvson, Basc S':lls Group Natonal nsttute of Educaton th Street, N.W. Washnton, DC Dr. Erc McWllams Program Manager Technology and Systems, TE Natonal Scence Foundaton Washngton, DC Dr. Scarva Anderson Executve Drector for Specal Development Educatonal Testng Servce Prnceton, NJ m Professor Keth Wescourt Stanford Unversty nst. for Mathematcal Studes n the Socal Scences Stanford, CA Dr. Bernard M. Bass Unversty of Rochester Management Research Center Rochester, NY Century Research Corporaton 4113 Lee Hghway Arlngton, VA m Dr. Robert Glaser, Drector Unversty of Pttsburgh Learnng Research and Development Center Pttsburgh, PA SS^SS^S^^T^sss^^ 11 MMmsnMMHRM JüKKBK'f."»vt Mh..- o :,'.-. ^ V,.'....,.--.. (.. -:'-: (.'' '

43 ;- ;,:,^gmaaay»bmg'a»^5w^ Page 10 Dr. Kenneth E. Clark Unversty of Rochester College of Arts and Scences Rver Campus Staton Rochester, NY ERC Processng and Reference Faclty 4833 Rugby Avenue Bethesda, MD Dr. Vctor Felds Department of Psychology Montgomery College Rockvlle, MD Dr. Henry J. Hamburger Unversty of Calforna School of Socal Scences rvne, CA Dr. Rchard S. Hatch Decson Systems Assocates nc Rockvlle Pke Rockvlle, MD Dr. M.D. Havron Human Scences Research, nc. Westgate ndustral Park 7710'01d Sprnghouse Road McLean, VA Dr. Lawrence B. Johnson Larence Johnson and Assocates, nc. 200 S. Street, NW, Sute 502 Washngton, DC '1 V Dr. Davd Klahr Carnege-Mellon Unversty Graduate School of ndustral Admn. Pttsburgh, PA Dr. Robert R. Macke Human Factors Research, nc 6780 Cortona Drve Santa Barbara Research Park Goleta, CA m: :_ ssss;

44 : ,.....,..,,... Dr. Andrew R. Molnar Technolopcal nnovatons n Educaton Natonal Scence Foundaton Washngton, DC Dr. Leo Munday Vce Presdent Amercan College Testng Program P.O. Box 168 owa- Cty, A Dr. Donald A. Norman Unversty of Calforna, San Dego Center for Human nformaton Processng La Jolla, CA Pare 11 Mr. Lug Petrullo 2431 North Edgewood Street Arlngton, VA Dr. Dane M. Ramsey-Klee R-K Research & System Desgn 3947 Rdgemont Drve Malbu, CA Dr. Joseph W. Rgney Behavoral Technology Laboratores Unversty of Southern Calforna 3717 South Gr-c Los Angeles, CA Dr. Leonard L. Rosenbaum, Charman Department of Psychology Montgomery college Rockvlle, MD Dr. George E. Rowland Rowland and Company, nc. P.O. Box 61 Haddonfeld, NJ Dr. Arthur. Segel Appled Psychologcal Servces and Scence Center 404 East Lancaster Avenue Wayne, PA

45 .,. - - : -;. :-: [-.: :- ; :. :. : : Pape 12 Dr. Benton J. Underwood Northv^estern Unversty Department of PsycholoRy Evanston, L Dr. Davd J. Wess Unversty of Mnnesota Department of Psychology Mnneapols, MN Dr. Kenneth Wexler Unversty of Calforna School of Socal Scences rvne, CA Dr. Anta West Denver Research nsttute Unversty of Denver Denver, CO Dr. Rchard Atknson Natonal Scence Foundaton Washngton, D.C. Professor Rchard D. Snow Department of Educatonal Psychology Stanford Unversty Stanford, Calf John H. Wolfe U.S. Navy Personnel Research & Development Center San Dego, Calf Dr. Wllam Mann nformaton Scences nsttute 4676 Admralty Way Marna del Ray, Calf Professor Derek Sleeman Dept. of Computer Scence The Unversty Leeds 2 England Dr. John Annett Deparvment of Psychology The Unversty of Warwck Coventry CV47AL ENGLAND -r.-k : :: '- < :- - ^r''' - '- -

46 '.; Page 13 «Mr. Samuel Ball Educatonal Testng Servce Prnceton, NJ ü ff 11 u Dr. Gerald V. Barrett Unversty of Akron Department of Psychology Akron, OH Dr. Ronald P. Carver School of Educaton Unversty of Mssour-Kansas Cty 5100 Rockhll Road Kansas Cty, MO Dr. Ruth Day Yale Unversty Department of Psychology 2 Hllhouse Avenue New Haven, CT Dr. Edwn A, Fleshman Vstng Professor Unversty of Calforna Graduate School of Admnstraton rvne, Calforna HumRRO Central Dvson 400 Plaza Buldng Pace Boulevard at Farfeld Drve Pensacola, FL HumRRO/Western Dvson Berwck Drve Carmel, CA ATTN: Lbrary HumRRO Central Dvson/Columbus Offce Sute 23, 2601 Cross Country dr,e Columbus, GA HumRRO/Western Dvson Berwck Drve Carmel, CA ATTN: Dr. Robert Vneberg HumRRO Joseph A. Austn Buldng 1939 Goldsmth Lane Lousvlle, KY l»ssg^^. :.,- s. '-^

47 l:'m;.->!s!&mt*vw*^mmmzw! ^^,.,.^>v,, ;,^ ,..:.;... ; :;-, ^, T,;,-. /,.^r^:j/ v,, ; ;! ;. B.>-yv'v--^-^ ^ ^l^- - ^-V.v- Pape 14 Dr. Arnold F. Kanarck Honeywell, nc Rdge Parkway Mnneapols, MN Dr. Roger A. Kaufman U.S. nternatonal Unversty Graduate School of Human Behavor Ellott Campus 8655 E. Pomerada Road San Dego, CA Dr. Steven W. Keele Unversty of Oregon Department of Psychology Eugene, OR Dr. Alma E. Lantz Unversty of Denver Denver Research nsttute ndustral Economcs Dvson Denver, CO Mr. Bran McNally Educatonal Testng Servce Prnceton, NJ Mr. A.J. Pesch, Presdent Eclectech Assocates, nc. P.O. Box 178 North Stonngton, CT Dr. Steven M. Pne Unversty of Mnnesota Department of Psychology Mnneapols, MN M H Mr. Denns J. Sullvan c/o HASC, Buldng 119, M, P.O. Box Los Angeles, CA Dr. Patrck Suppes Stanford Unversty nsttute for Mathematcal Studes n the Socal Scences Stanford, CA '"".;-;^^':v;vv\-;-.v--^--'- : J'^='''-"^>;'' : -'^^33TVf^^tL-v'::' ^^y* wamjräsraw «SKWEWTD-M -^^rr >. v.:-.-..v.m.vjt- -- ^p^^mt^h^*^^

48 -.... ' --; Pa/re 15 Dr. K.W. Uncapher Unversty of Southern Calforna nformaton Scences nsttute ^676 Admralty Way Marna Del Rey, CA Dr. Carl R. Vest Battelle Memoral nsttute Washngton Operatons 2030 M Street, N.W. Washngton, DC S Dr. John J. Collns Vce Presdent Essex Corporaton 6305 Camnto Estrellado San Dego, CA Mr. Charles R. Rupp Advanced W/C Development Engneerng General Electrc Company 100 Plastcs Avenue Pttsfeld, MA f m m m * 1 E