a BOLT B E R A N E K AND NEWMAN 'NC a W- CONSUTNG DEVEOPMENT RESEARCH / BBN REPORT NO. 3346 <^y to o THE EFFECTVENESS OF AN NTERACTVE MAP DSPLAY N TUTORNG GEOGRAPHY CM t Allan Collns Marlyn Jager Adams Rchard W. Pew 1 August 1976 Sponsored by Offce of Naval Research and the Advanced Research Projects Agency Approved for publc release; dstrbuton unlmted, Reproducton n whole or n part s permtted for any purpose of the Unted States Government. BOSTON WASHNGTON CHCAGO HOUSTON LOS ANGELES OXNARD
M >.v,-., ;'..:.\:.. The Effectveness of an nteractve Map Dsplay n Tutorng Geography ft;! Allan Collns Marlyn Jager Adams Rchard W. Pew Bolt Beranek and Newman nc. Cambrdge, Massachusetts 02138 Contract No. N00014-76-C-0083, effectve September 15, 1975. Expraton Date, September 30, 1976 Total Amount of Contract - $187,000 Prncpal nvestgator, Allan M. Collns (617) 491-1850 Sponsored by: Offce of Naval Research Contract Authorty No. NR 154-379 Scentfc Offcers; Dr. Marshall Farr and Dr. Joseph Young and OJ4# Advanced Research Projects Agency ARPA Order No. 2284, Amendment 5 Program Code No. 61101E NHR The vews and conclusons contaned n ths document are those of the authors and should not be nterpreted as necessarly representng the offcal polces, ether expressed or mpled, of the Advanced Research Projects Agency, the Offce of Naval Research, or the U S Government. Approved for publc release; dstrbuton unlmted. Reproducton m whole or n part s permtted for any purpose of the Unted States Government. S
ss? o,^. ^;!^SÖ^^a^äÄ-'^/S,^,o MnMMRsnn SLCURT Y CU ASSH K" AT ON OF TMT. P A.,'. E 'll A d /)(Ul l.lllrrrd), H rom NUMB L R REPORT DOCUMENTATON PAGE Techncal Report No. 2 / ^»(rtrffcr"*""'''"'*' 2. GOVT ACCESSON NO.-mrafrwKW».,»»»"»»«BsaBM^^j, JBE EFFECTVENESS OF AN NTERACTVE J1AP "TDSPEAY N.TUTORNG GEOGAPHYJ ^ 7. *UTHOR(Sl wwwmawwg'gwjgjjwj"^^'**^ HKA) N.STUCTONs ; ()HK COM'.-. M. -1 MM BBN Report No. 334 6 S. CONTRACT OR GRANT NUMBERS) -.. Al an/coll ^ ns, Marlyn Jager/Adam^, :/* Rchard/Pew l. PERFORMNG ORGANZATON NAME AND ADDRESS t. Bolt Beranek and Newman nc. / 50 Koulton St., Cambrdge, Ma. 02138 CONTROLLNG OFFCE NAME AND ADDRESS Personnel and Tranng Research Programs Offce of Naval Research (Code 458) Arlngton, " xnqton, Va. 22217 MV^röNrrffrt-wo-A&ENCXNAME.fc^AODFess^/.d/Me''/fO'n ControUngOffce) MttMrnanM^ag.««. N00014-76-C-0083 10. PROGRAM ELEMENT PROJECT TASK AREA & WORK UNT NUMBERS 61153N RR042-04-01 NR154-379 ^> s / 15. SECURTY CLASS, (of thn refptrtj f Unclassfed 18 0. DECL ASSFC ATON/ DOWNGRADNG SCHEDULE 16. DSTR JTON ST ATEMENT ^o/ l/lls /?epor(>.^ /-:. \7,Q\lt\*U T rto^ sj,*xzal*&'^^'k^a^umt^^ f rom l e P on ) 7 ^*fc(3»(vj«ö= ^^«**«maw.,vwjäjk^.^»,; W^MBltat Ul 'V ft *h y/ <? wt*w. JKCT «ml"^' 7 MOWMMMRS WHrwwwaMWryw To^b^^subm^ted^'tö^trüurttal^of^"^^ Psychology s f *4a' s ] S. KEY WORDS (Conthue on reverse äe/neeesary and fentfy by block nu Educaton, Tutorng, Computer-asssted nstructon. Learnng, Teachng, Educatonal Psychology, Evaluaton. 20. ABSTRACT ^Contnue on reverse sde f necessary and dentfy by block number) *"The purpose of ths study was to evaluate the teachng effectveness of dfferent aspects of the SCHOLAR CA system. The experment compared how well students learn usng SCHOLAR wth (a) the nteractve map dsplay of Map-SCHOLAR (b) a statc labeled map, and (c) an unlabeled map. The results of the experment showed that the students learned sgnfcantly more wth the nteractve map dsplay, than wth ether the labeled - DD t j 73 14^3 EDTON OF 1 NOV 65 S OBSOLETE -V.; /*)// /- / " ^ ') -_- ' V-.- r SECURTY CLASSFCATON OFTHSPAGEfBAen Dala - nfrt-jl
f. URTY CL Ar.Sl'lC Al ON OF THS PAGE (Sh-u JhaTnlrrr) vz: (20) 's -a..,js- lü.cjl... ^-7v\_cj(_-C-j map,^ A new method called backtrace analyss was used to assess the effectveness of specfc aspects of the tutorng strategy and the map system used n the experment, /o~y -,r.ft, ;.- -«ss "::. d SECURTY CLASSFCATON OF THS PAGE(9ken Data Entered)..:; ' ' *::: ^... n a,«.., «wn.,.tu «j.n.. awg
1 1...!-,.^ ^ ^ '':L..;.:-....,: :.,.. :,, ^., : ',:',:j../.::,^ ',- «/ 'k:r,,p^-^-' &. '','? :.1,.^..,:\:.UJ^^.:::^ r-.>-\-,-:/,:: :;., :.:: ::. - Lh-^^^^: ^--V^v^.l :j:n:f*-u '." ' - ' 1 The Effectveness of an nteractve Map Dsplay n Tutorng Geography Allan Collns Marlyn Jager Adams Rchard W. Pew Bolt Beranek and Newman nc. 50 Moulton Street Cambrdge, Mass. 02138 Ths research was sponsored by the Personnel and Tranng Research Programs, Psychologcal Scences Dvson, Offce of Naval Research, under Contract No. N00014-76-C-0083, Contract Authorty dentfcaton Number, NR 154-379. We would lke to thank Nelleke Aello, Susan M. Graesser and Barbara N. Freeman for programmng the system and carryng out the experments and data analyses descrbed. U:
*: r ;" ' 8Ü l ABSTRACT The purpose of ths study was to evaluate the teachng effectveness of dfferent aspects of the SCHOLAR CA system. The experment compared how well students learn usng SCHOLAR wth (a) the nteractve map dsplay of Map-SCHOLAR (b) a statc labeled map, and (c) an unlabeled map. The results of the experment showed that the students learned sgnfcantly more wth the nteractve map dsplay, than wth ether the labeled map or the unlabeled map. A new method called backtrace analyss was used to assess the effectveness of specfc aspects of the tutorng strategy vv S and the map system used n the experment. : ;; S-: '' :!.1^^..;r.'-:.^;>^:,'-,^::^-:;v.v.v-._.. > -^ _..._...,..._._.._.._............._._.... '..._.._._ -...... _. _ -.-..^.-^ -.- - w.r. y.':^ ' _^^ '
3 : '.^ ' ';,:. ;/, :V^. :.^-v. ^ ;-,.- 0.:..,.;,..;.. ;.-.- :: <:-';; :.'' ". >jv -V:': ^ J'r.M'-'-.X-^-'^:.-'vL^.;- NTRODUCTON n developng SCHOLAR, Carbonell (Carbonell, 1970; Carbonell and Collns, 1973) took a frst step toward a computer asssted nstructon (CA) system that s capable of conductng general tutoral dalogues wth students. n SCHOLAR, knowledge s not stored as text, but n a precselystructured semantc network of nterrelated facts and concepts (Qullan, 1968; Collns and Qullan, 1972a). Every concept used to descrbe a gven concept can tself be descrbed elsewhere n the network. Thus, n a nontrval sense, the program can understand the concepts t uses. SCHOLAR also has dfferent subroutnes that use the structure of the network to formulate questons for the student, evaluate hs answers, answer hs questons, make nferences and computatons, select new topcs for dscusson, etc. The attempt r to structure nformaton lke a human knowledge, so that the program can use ts knowledge as flexbly as a human tutor does. Because of ts nherent flexblty, the system can be extended n a varety of ways. For example, SCHOLAR has been modfed to teach dfferent knds of knowledge (Collns and Grgnett, 1975) and to use dfferent teachng strateges (Collns, Warnock, and Passafume, 1975). The educatonal queston s whch of the possble ways of varyng SCHOLAR ncrease ts teachng effectveness. Ths
9 1..:.-.;:.',....--v/.;.:.'':... ;,,. '...V.1... queston s beng nvestgated by runnng systematc expermental evaluatons of dfferent system confguratons and dfferent teachng strateges (Collns, '\97 \). The experment reported here was desgned to test the utlty of the map capablty recently added to the orgnal SCHOLAR program for teachng geography. The Map-SCHOLAR system was developed to ntegrate the tutorng of graphc nformaton wth verbal nformaton. n vew of the evdence that pctoral nformaton may be remembered more easly than verbal nformaton, (c.f. Bower, 1972; Pavo, 1971) we expected ths capablty to ncrease SCHOLAR'S teachng effectveness. Map-SCHOLAR can dscuss wth the student dfferent maps that change dynamcally accordng to the context of the s dscusson. To do ths, a graphc structure was created whch parallels the structure n the semantc network. The elements n the map dsplay can be referred to ether by ther name, or by pontng to them, or both. Map-SCHOLAR both asks and answers map-related questons and provdes relevant map nformaton when the student makes a mstake. n short, Map-SCHOLAR has all the capabltes of the orgnal SCHOLAR wth the addton of the map capabltes. Fgure 1 llustrates some of the varety of nteractons possble wth Map-SCHOLAR. The frst panel llustrates how Map-SCHOLAR asks map-related questons. """^^' ^^vwummma^mma^tmsvkmw^-lst^st
SX; ;',\..;.o'v';-.,,.'.' '.:. /'^^^^.^.^v'-.-/..'r'':^.v' ^U:^k.l;;^.'!^.,-^':.' :..;.-.;> -),! ' -.f'-'.-^ -v.rv-t' NOW LET'S SEE WHAT YOU REMEMBER ABOUT THE CTES OF BRAZL. PLEASE NAME THE PRNCPAL CTES OF BRAZL NOW BLNKNG ON THE DSPLAY SCREEN. *MANAOS SAO PAULO AND LMA* HANAUS AND SAO PAULO ARE CORRECT. LMA S N PERU, NOT N BRAZL. THE PRNCPAL CTES OF BRAZL ARE BLNKNG ON THE SCREEN. RO DE JANERO S N SOUTHEASTERN BRAZL. T S ON GUANABARA BAY AND THE ATLANTC, BRASLA S N CENTRAL BRAZL. T S FAR FROM THE OCEAN. RECFE 0 DE JANERO ORTO ALECRE Fg. 1 nteractons wth Map-SCHOLAR '& V: ' ' _""*
.. PLEASE r TO THE FOLLOWNG CTES OF THE AMAZON N THE ülder M WHCH THEY APPEAR BELOW: MANAOS, BELEM, AND QUTOS DON'T FORGET TO LET ME KNOW WHEN YOU ARE FNSHED, PLEASE REMEMBER TO TYPE.AN ASTERSK * FOLLOWED BY A CARRAGE RETURN. ** MANAOS S CORRECT. YOU DD NOT PONT AT BELEM AND QUTOS CORRECTLY, THE CTES OF THE AMAZON ARE BLNKNG ON THE SCREEN. ^ a : /,; ' ; ' m 1 Fg. 1 nteractons wth Map-SCHOLAR (cont) ^Lj.--k y^s.'jüu*
mmmm. J:WJ.J *. 1! 1.' 4 LET'S BEGN BY TALKNG ABOUT THE TOPOGRAPHY OF VENEZUELA. THE PRNCPAL LAKE OF VENEZUELA S BLNKNG, PLEASE NAME T. * DON'T KNOW* THE PRNCPAL LAKE OF VENEZUELA S BLNKNG ON THE SCREEN. E LAKE MARACABO S N NORTHWESTERN VENEZUELA. THE PRNCPAL MNERAL S OL. *S THS COUNTRY COLOMBA?* YES, THAT'S COLOMBA *WHERE S THE CAPTAL OF VENEZUELA?* CARACAS S BLNKNG. m Fg. 1 nteractons wth Map-SCHOLAR (cont) n HP _-, MP^^SSäSfW.«nllMMHHmWW m «-»«unllm llmllamlmmwmmmmlllmw^^
E evaluates the student's answers, and corrects hs mstakes. Frst, the dots ndcatng the locaton of the ctes appear unlabelec on the map of Brazl and start blnkng. Then SCHOLAR asks the student to name the blnkng ctes. When B he responds, t tells hm whch answers were correct and prnts ther names on the map. Because he erroneously gave Lma as an answer, Map-3CH0LAR ponted out that Lma s n Peru and showed where Lma s on the map. Thus, Map-Scholar gves both verbal and vsual feedback to help the student learn the materal. The second panel shows how context affects the map dsplay. The topc s the Amazon, whch appears n ths t context on the map of Brazl, the overall topc. The queston asks the student to pont at the ctes along the Amazon n the order named. The dots for the three ctes frst appear unlabeled, and only after the student has answered do the labels appear. Unlke the frst map, ths map shows only the ctes on the Amazon and ther relaton to the Amazon. There s also expanson of detal so that a cty lke Belem, whch was not mportant enough to be ncluded among the ctes of Brazl, s ncluded wth the subtopc of the Amazon. Because the map changes dynamcally as the context changes and as the student nteracts wth the system, the student's attenton s focused on the relevant nformaton, and questons can be posed n a vsual form not E possble wth a statc map dsplay. 7 ^ ' ^ V'v"^*,. ;V\V:. ;-:rv;: "'TV..;-"' " '' ^ ; >:.-V-..-~..../;- -. :..* -... ^.^v-- -.../ JW^:--..,'-^ ^-><.. ->,-...:.-. : j-r:....^,-^^..- BMW«KS6at^.--3S.1)^l:C(.ü'W^
Lü-l^ "' Ja '-- The thrd panel shows the system's potental for tutoral nteracton. When the student ddn't know about Lake Maracabo, Map-SCHOLAR showed t on the screen and added the related verbal nformaton about the ol there. Ths example also llustrates some of the ways n whch the student can ask Map-SCHOLAR to clarfy or amplfy the nformaton gven (Collns and Warnock, 1974). For the frst queston the student both ponted at and named Colomba to ask f t s the country near Lake Maracabo. For the second queston, the student verbally asked where the captal of Venezuela s, perhaps to fnd out how far away t s. SCHOLAR fgured cut semantcally that the captal 3 Caracas, and then vsually showed where Caracas s by blnkng t (t s the double square). These examples llustrate some of the power for tutoral nteracton that can be obtaned by a close ntegraton between semantc and vsual knowledge. n order to test the utlty of the map system for teachng, we conducted an experment n whch each student learned about a dfferent country under one of three condtons: one condton used SCHOLAR on the map system; the second condton used SCHOLAR on a non-graphc termnal, but the student could look at a labeled map of the country; the thrd condton was lke the second, except that the student was gven an unlabeled map Students' learnng for each of the three knds of tranng sessons was measured by m 8»^«SWH^«^^
comparng ther scores on a pre-test to those on a post-test gven three days after the last tranng sesson. A second goal of th; experment was to nvestgate how specfc aspects of the tutoral dalogue affect students' learnng. To study ths queston, we developed a technque called backtrace analyss. The technque nvolves markng each pece of nformaton that s dscussed accordng to the knd of exchange nvolved (e.g., a queston requrng a pontng response vs. a namng response). By comparng ths data to the student's answers on the post-test, t s possble to dentfy the knds of tutoral nteractons that most strongly nfluence the student's learnng. METHOD v.j : Subjects. The ntal group of subjects ncluded nne hgh school students. The study was replcated wth nne unversty ^udents. All subjects were volunteers and were pad for ther servces. Desgn. There were three expermental condtons: a Map-SCHOLAR condton, a Labeled Map condton, and an Unlabeled Map condton. The Map-SCHOLAR condton was run on an mlac graphc termnal wth the screen dvded between maps and verbal communcatons as shown n Fgure 1. The student could nput questons and answers by a keyboard and an electronc ponter (a "mouse"). The Labeled and "':mm ' " ^'' ""ywgfflpwwwajbm
-'..:: : V '. Unlabeled Map condtons were run on a keyboard termnal usng a non-graphc verson of SCHOLAR called Tutor-SCHOLAR (Collns, Warnock, and Passafume, 1975). The two versons of SCHOLAR were dentcal wth respect to both teachng strategy and nformaton n the data base, except that Map-SCHOLAR handled all locaton-related questons n terms of the map, whereas Tutor-SCHOLAR handled them verbally. n the Labeled Map condton, subjects were gven a paper map whch marked all the places (names and locatons) ncluded n the Map-SCHOLAR data base. n the Unlabeled Map condton, subjects were gven copes of the same maps, omttng the names of the places that were marked on the maps. For both of these condtons, students were nstructed not to mark on the maps. The pre-test, post-test, and the fnal questonnare were gven n paper and pencl format. Procedure. Each student partcpated n a prelmnary sesson, three tutoral sessons, and a post-test sesson. The frst purpose of the prelmnary sesson was to admnster the pre-test. The pre-test measured the student's pre-expermental knowledge about the nformaton to be tutored, and conssted of 20 basc questons about the geography of each of the three relevant countres: Argentna, Brazl, and Venezuela. A secondary purpose of the pre-test was to ascertan that no subject was nordnately famlar or unfamlar wth any one of these 10 S^SJ^SÄW^ÄKoss-wcea»! ^
j «L«m.Ku n..mw.»..ll..'.."gttr f three countres, snce such nequaltes n pror knowledge would confound measures of teachng effectveness. After havng completed the pre-test, the student was gven a bref, ntroductory lesson on a fourth country, Chle, usng Map-SCHOLAR. The; purpose of ths lesson was to famlarze the student wth the system and ts capabltes or, more specfcally, wth the knds of questons he would be asked, the knds of answers that were expected of hm, the knds of questons he could ask of SCHOLAR, the use of the keyboard and the ponter, and the methods by whch he could correct hs nput errors. t;: The tutoral phase of the experment conssted n three, two hour sessons, admnstered on consecutve days. Durng these sessons, each student learned about one country n the Map-SCHOLAR condton, one n the Labeled Map condton, and one n the Unlabeled Map condton. Each lesson lasted for one hour. After the student had receved one lesson on each of the three countres, the seres was repeated. The combnatons of countres and teachng modes were counterbalanced and ordered accordng to a 3 x 3, confounded, factoral desgn (Wner, 1971, p. 6^6). The fnal sesson was conducted three days after the last tutoral sesson. n ths sesson, the student took the post-test and completed a questonnare on those aspects of the lessons that he had found most and least helpful. n
_..._._. _._ :......,.,..._^..,-' The post-test was dvded nto three parts. The frst part conssted of 36 basc questons (ncludng the 20 that had been on the pre-test) about each of the three countres. For the second part of the post-test, the student was gven a map of each of the three countres and asked to label the geographcal features ndcated. The thrd part of the post-test conssted of 32 more dffcult questons about each of the countres. Backtrace Analyss. n order to assess the value of specfc aspects of the tutoral exchange, we developed the technque of backtrace analyss. Ths technque nvolves markng each entry n SCHOLAR'S database wth respect to the way n whch t s treated durng a gven tutoral sesson. Ths nformaton can subsequently be retreved, enablng us to evaluate the effectveness of SCHOLAR'S varous nteractve capabltes from the probabltes wth whch they result n correct answers on the post-test. More specfcally, each tem that was dscussed n a gven sesson was tagged wth nformaton concernng (1) the temporal order, (2) the context, and (3) the tranng event n whch t arose. For purposes of the backtrace analyss, the tranng events were classfed as follows: a) True-False Correct - SCHOLAR presents a true-false queston whch the student answers correctly. SCHOLAR tells the student he s correct and moves on to new 12
- -. >*^ M ll-ml ftmu. LL'g ;J1 m a! m 1 E nformaton. b) True-False Error - SCHOLAR presents a true-false queston and the student answers ncorrectly or pleads gnorance. SCHOLAR ponts out the correct answer and goes on. c) Name Correct - The student correctly names a geographcal feature(s) n response to SCHOLAR'S request. Each answer among a set of answers s tagged ndvdually. Ths category subsumes what and where questons as well as fll-n-the-blanks and namng requests by SCHOLAR. d) Name Error - The student ncorrectly names or fals to name a geographcal feature when questoned by SCHOLAR. e) SCHOLAR Error Correcton - f the student completes a fll-n queston erroneously, SCHOLAR nfers the bass of the student's error and then presents new nformaton to dstngush between the student's answer and the correct answer. f) SCHOLAR elaboraton - f the student msses a queston, SCHOLAR presents related nformaton at the same level of mportance (See Fg. 1). The related materal s tagged as an elaboraton. g) Student Queston - nformaton s ntroduced as the result of a queston the student asks of SCHOLAR. n addton to the above, there were several categores of 'tranng events whch occurred only n Map-SCHOLAR. SCHOLAR treated these events lke fll-ns, but they were 13 VfSjSröK rjtpftqrw*«*»" * "
dstnctvely marked for purposes of the backtrace analyss: h) Label - SCHOLAR asks the student to name those features of the map that are blnkng, ) Pont - SCHULAR asks the student to pont to the specfed geographcal features on the map. j) Label and Pont - SCHOLAR asks the student to name and pont to a specfed set of geographcal features. RESULTS AND DSCUSSON The pre-test scores were examned usng a 3 X 2 (Countres X Groups) repeated measures analyss of varance (Wner, 1971, p. 518). The only sgnfcant effect was due to groups, as the college students generally scored hgher than the hgh school students. The number of correctly answered questons, out of the possble 20 per pre-test, ranged from 1 to 11 (medan = 4.67) for the college students and from 0 to 5 (medan = 0.64) for the hgh school students. Nether the man effect of countres (F(2,32) = 2.62, p>0.05) nor the nteracton between countres and groups (F(2,32) = 1.36, p>0.05) approached sgnfcance. nasmuch as none of the subjects knew much about any of the countres n advance, the dl ce between pre- and post-test scores should prov le far estmate of SCHOLAR'S teachng effectveness. Moreover, snce the subjects' pror knowledge seemed to be evenly dstrbuted 14
across countres, the relatve teachng effectveness of the three condtons could be estmated through drect ;: comparsons of the correspondng pre-test/post-test dfference scores. Wl :. :::. The average ncrease n the number of correct responses from the pre-test to the post-test s shown n Fgure 2 for each teachng mode. These dfference scores were analyzed accordng to a 3X3 (teachng modes X countres) confounded factoral desgn. Whereas nether the effect of countres (F(2,28)<1.0) nor the nteracton between countres and teachng condtons (F(4,28) = 2.08, p>0.05) was sgnfcant, the effect of tranng condton was strongly sgnfcant (F(2,28) = 6.05, p<0.01). Accordng to a Newman-Keuls test (p<0.01), the Map-SCHOLAR condton resulted n sgnfcantly hgher post-test scores than the Labeled Map condton whch, n turn, resulted n sgnfcantly hgher scores than the Unlabeled Map condton, Separate analyses of the three parts of the post-test ndcated that much of the effect of teachng modes occurred n the part of the test consstng of map labelng questons (F(2,28) = 14.09, p<0.001). However, a pronounced effect of teachng mode was also obtaned for the easer, non-map questons n the frst part of the post-test (F(2,28) = 5.85, p<0.01). Although the 15 MM^MVBMNlHMBBamaNHnBBBB ^^^^^^...:...:...' ^.;.,,%';-:^v:,--;.;.-:...; ;. t.-- --.j--^-^a..;-.
fe - ;,: \ : <; -.: : : : :...".-....,. : -;;S ' ;-.: 'v-->,.".v-\.j..^ -.';: :.....,. -:.':-., _,',';. J-S CM LL) CD o ~ o < LL. < ^ CO 5 < Q Q 2 LL! O LL) - o h- LU Z X 3 h- JJ 11 o z < H D Q z o Q O Ü ro ^ m CVJ o CV n n :> ls31-ls0d QNV ls31-3dd N33M13a 3d03S 33N3a3JJa 39Vd3AV 16 ^--v^.'.v^-;:. v^^-.,r.;'-\^^^:-;^^^-!.---^^-v?;r-^/^"^r-;-^:\v-y-^.7(.^^
l/^^-^-^-..^.;:r';j.v-:-".v '...-; -_^.1 v ^ '. ' ' ''" ".'' scores on the more dffcult questons n the thrd part of the post-test were too nosy to yeld any sgnfcant effects or nteractons under analyss, the same trend was apparent. n short, post-test scores were consstently hghest n the Map-SCHOLAR condton and lowest n the Unlabeled Map condton. These results ndcate that the map system not only helped students learn the nformaton necessary to answer the map questons n Part 2 of the post-test, but also to answer the verbal questons n Parts 1 and 3 of the post-test. An mportant queston s whether the beneft of the map system extended only to verbal nformaton that was explctly about locatons, or whether t also extended to non-locaton nformaton, such as the clmate or terran of a place. Clearly, one would expect the map?;' ' system to help students learn locaton nformaton better, but there are two reasons why the map system mght help students learn non-locaton nformaton better as well. Frst, f map nformaton showng where a place lke Manaos s located helps the student remember Manaos better, then non-map facts about Manaos, such as ts clmate, may be remembered better. Ths s because the best way to learn somethng s to relate t to nformaton already known (Collns and Qullan, 1972b; Norman, 1973). Second, f a student sees that Manaos s on the Amazon, then Manaos' clmate can be related to any 17
'. --. ; '>' r : )^-u''?-.', pror knowledge about the clmate of the Amazon (e.g. that the Amazon flows through jungle). Thus, even non-map nformaton may be better remembered n a vsual context. Ths dea was tested wth backtrace analyss by separatng the questons durng tranng nto map questons and non-map questons, dependng on whether the questons were posed vsually by the map system. Then the percentage correct on the post-test for the two types of presentaton durng tranng were plotted (see Fgure 3). For map questons, as expected, students learned sgnfcantly more wth Map-SCHOLAR than wth ether the labeled or unlabeled maps. But for non-map nformaton there were no sgnfcant dfferences, and students even dd slghtly better n the Unlabeled Map condton. Thus, these data suggest that the major beneft of the map system s n learnng nformaton about specfc locatons. Backtrace analyss was also used to nvestgate the effectveness of repeatng questons, dependng on whether the student answers correctly or ncorrectly. Fgure 4 shows the percentage of correct responses to each tem on the post-test as a functon of how frequently the students were correct or wrong on that tem durng tranng. The ncreases n the curves show 18 '. ;.;:.,. ' -,-.,,.>. : -. ^.- ^"-^
mmam um «mmnnm.nmüm.wmnnjum' l CO o: CJ3 g H Q z o ÜJ < O UJ < cc O c^ L_ Z o: o UJ r: ^^ H UJ c^ ^ UJ o CD Z z < 01 \- Ü z or a z o o z o CO ^ z: a. < S31-S0d NO 103dd03 ln33d3d lo: 19
o 2 O 0 - ^ h- tfc o 2: -z. <r r) rr u_ h- < 0 </) 2 < cc: 1-3 ÜJ rr H 0 f~ or C/) LL O Lü * 0. -z. - LÜ 1 " S - 1 - ^ " HO U- 2H O O h O a: 0 cr 0 o: 0 j m "Z. ÜJ _) O -^ CC Lü a. 1S31 SOd NO ld3üyoo NBOüSd
: ;;....-...: ''-:...,....; '.^^A^'.jv.hL.,j/.s.:;.-.^.^..:>j;;,' that the more frequently a student answered any tem correctly, ehe more lkely he was to recall t on the post-test. The separaton of the curves for dfferent numbers of errors shows that the more frequently an tem was mssed, the less lkely t was to be recalled on the post-test. Ths smply reflects the fact that the tems that were more dffcult to learn were lkely to be mssed more frequently. The concave shape of the curves ndcates that the repeated presentaton of a correct tem has a decreasng effectveness. The mplcaton s that, as far as possble, tranng tme should be allocated to those tems that the student has correctly answered least often. When students mssed tems n answerng a queston, SCHOLAR provded addtonal elaboraton about some of the tems mssed. For example, n Fgure 1 when the student dd not know Lake Maracabo, SCHOLAR mentoned the ol n OTT! H ll m ü 81 $' :":;J there as an elaboraton about Lake Maracabo. The backtrace analyss showed that percent correct on the post-test ncreased from 3^% when there was no elaboraton of an tem durng tranng to 47$ when there was one elaboraton. Ths ncrease s sgnfcant (t = 4.01,p<0.01), ndcatng that elaboraton does help students to learn the materal better. After one elaboraton, the percent correct stablzes, ndcatng that further elaboratons are of lttle beneft. 21 f
We used a varaton of backtrace analyss to determne whch knds of map questons are most effectve for learnng. n the map system there were three dfferent knds of map questons that mght be asked: (1) pontng questons, where SCHOLAR mentoned one or more places and asked the student to pont at them (2) namng questons, where SCHOLAR blnked one or more places and asked the student to name them, and (3) pontng and namng questons, where SCHOLAR asked the student to name a set of places, such as the rvers n Brazl, and pont to them n the order named. Fgure 5 shows the percent correct on the second occurrence of a map queston about any tem as a functon of the type of queston that was asked on the frst occurrence of that 'em. There were not enough data for namng questons, so they are not shown. The column totals ndcate that students dd better on pontng questons than on pontng and namng questons, as would be expected because pontng questons are easer. However, the row totals show that students dd better on the second queston f the frst queston had requred both pontng and namng than f t had requred only pontng ofl) = 4.75,p<0.05). Evdently, students learn more from pontng to and namng a locaton than from Just pontng to t. 22
^^^rt Vfl^^B? wy«-^ - Vlfv:,.^ ^-r.t-f::'. q 6 v a. m SB y; f CO Q. >- H CD 2 < QjCD a: ^ Lü or to Q CO o o h-hoco Lü Lü LL 3 U. O LU üü X ÜL < UJ U z UJ a: 80 H- O 2 Z UJ O O O CC LU UJ CO o z o a. O $5 ro S5 05 o z 1- z o a. n S5 0^ 5 o a o z z z <p <? z o a. 0 s ro n O V a. 4 L, o UJ z Z QJ OQ:»-^ UJO ^^ OO OlO a: UJ Q.U. > KZ O 23 ^v-t-q :^ rx-?r:;z \r:.vc^>r.xaamsa!^^ ': - :. :.J-C ::% ': ^
CONCLUSON The experment showed that students learned sgnfcantly more wth the nteractve map dsplay than wth ether a statc labeled or unlabeled map. The advantage of Map-SCHOLAR cannot be attrbuted solely to the ablty of the student to locate places spatally, snce the Labeled Map condton allowed the student to dentfy places just as effectvely. The advantage of Map-SCHOLAR also cannot be attrbuted to novelty or some other generalzed facltaton effect, because, as backtrace analyss showed, the effect was specfc to locaton nformaton and dd not carry over to non-locaton nformaton. The advantage therefore must have been due manly to the dynamc aspects of Map-SCHOLAR and ts ablty to focus the student's attenton on the relevant map nformaton. The experment also demonstrated the usefulness of the backtrace analyss technque for evaluatng CA systems. Backtrace analyss s not dependent on the type of nformaton beng taught, and s thus transferable to CA systems other than SCHOLAR. Of course, the specfc tags used to mark the data would change, dependng on the dfferent teachng strateges and tranng events that are beng evaluated. The ablty to perform fne-gran analyses of the effectveness of dfferent teachng 24
f...'...... "..-., '-^.'. ':'.-. " - ;- -. % '- ' : '. '.' * -r. : ' - -, ^-.-. " ;';>. " -.'.,..,,_...,,...,........ strateges s a valuable tool for future educatonal research. -f; P t }\ ^! p ^. ra»a 1 «fn 11 Eäfe 1^ 25
m m ^W5^!55W,»^a^»W-?«^^K»^^^ '-' ^ K *,1^^TW^?,ä y3^^p^^ ppp^h3^p!3 ÜÜ,..,.y;:.. /.-v'.... : : -.:,.,^,,;. v..v:.: :..-!v :v.',-.ä*,;::v;,. AMmMMMsä REFERENCBS Bower, G.H. Mental magery and assocatve learnng. n L.W. Gregg (Ed.), Cognton n Learnng: and Memory. New York: Wley, 1972. Carbonell, J.R. A n CA: An artfcal ntellgence approach to computer-aded nstructon. EEE Transactons on Man-Machne Systems. 1970, MMS-11, 190-202. Carbonell, J.R. and Collns, A.M. Natural semantcs n artfcal ntellgence. n Proceedngs, of the Thrd nternatonal Jont Conference on Artfcal ntellgence, 1973, 3^-351. Collns, A. and Grgnett, M.C. ntellgent CA. BBN Report Mo. 3181, 1975. To be submtted to nstructonal Scence. Collns, A.M. Comparson of two teachng strateges n computer-asssted nstructon. BBN Report No. 2885, 1974. Submtted to Contemporary Educatonal Ps.ychol og^. Collns, A.M., and Qullan, M.R. How to make a language user. n E. Tulvng and W. Donaldson (Eds.), 26
>:. «, ',-....w.... '..,,^ Organzaton of Memory. New York: Academc Press, 1972a. Collns, A.M. and Qullan, R.M. Experments on semantc memory and language comprehenson. n L.W. M m Gregg (Ed.) Cognton n learnng and memory. New York: Wley, 1972b. m MM Collns, A.M. and Warnock, E.H. Semantc networks, BBN Report No. 2833, 1974. ml r.» Collns, A.M., Warnock, E.H., and Passafume, J.J. Analyss and synthess of tutoral dalogues. n G.H. Bower (Ed.) The psychology of learnng and motvaton, Vol. 9 New York: Academc Press, 1975. Norman, D.A. Memory, knowledge, and the answerng of questons. n R.L. Solso (Ed.), Contemporary ssues n cogntve psychology: The Loyola symposum. New York: Halsted Press, 1973. m $....jj 1 g m E :./: :' :[-::^^V$^^ Pavo, A. magery and verbal processes. New York: Holt, Rnehart, and Wnston, 1971. Qullan, M.R. Semantc Memory. n M. Mnskey (Ed.), Semantc nformaton processng. Cambrdge, Mass.: MT Press, 1968. 27
'äkwwkwwsmbw«^8^ Wmmä::^ ;.:;:,. :>.. S':rf, ä : :;.;w,&;..._ :, :,...;..V,\.,; : ;. ;,,,,,.,. :._,,,,,, r**~4 Wner, B.J. Statstcal prncples n ex^ermental desgn. New York: McGraw Hll, 197t, 28
'- %, " :. ;... ;- :. -v':-:.:' -.;... ^ -'.:.:^0,.-'.:."--^-^-.-^; : --:- ; "..,' 1 W-:'-, ; '/--,'v.._,-, c,.-;-'^.%;;:'.:;;.' c^vv'-vu.- 1 '- 1 ;'.^.:-^^^ ^'^ ^^^^j:-^.^;vvv::/^^^o> ^^ ;rvvl ^ : ^ ; -J;^ ;:v; -v^/^/:^^;^v ;;'-1 ^: v.",. -.;.' ; ;; '^ 1 ^ v v ^:^^ ^;'v.-.^; ^-^ H DSTRBUTON LST Dr. Marshall J. Farr, Drector Personnel & Tranng Research Programs Offce of Naval Research Arlngton, VA 22217? as Sa TO mm Drector ONR Branch Offce 495 Summer Street Boston, MA 02210 ATTN: Dr. James Lester Drector ONR Branch Offce 1030 East Green Street Pasadena, CA 91101 ATTN: E.E. Gloye Drector ONR Branch Offce 536 South Clark Street Chcago, L 60605 ATTN: Dr. Charles Davs Dr. M.A. Bertn, Scentfc Drector Offce of Naval Research Scentfc Lason Group/Tokyo Amercan Embassy AP0 San Francsco 96503 Offce of Naval Research Code 200 Arlngton, VA 22217 H W 1 Assstant Deputy Chef of Naval Personnel for Retenton Analyss and Coordnaton (Pers 12) Room 2403, Arlngton Annex Washngton, DC 20370 Dr. Lee Mller Naval Ar Systems Command AR-413E Washngton, DC 20361 Commandng Offcer Naval Health Research Center San Dego, CA 92152 ATTN: Lbrary AS v.5>:?üfts-bny'/p;f-"'f.'^''.".':r"v..--.::~:.t.'!^'k., swcä:
ük :, :^.. - - :. : '.< ::. -...^.,..,.....- ' ' '- : ' ' ' -.^ ' ^ * :' ' '' '!S!p«$!ep8!S!^^yt : v''^ ^ : ". ' Page Drector, Navy Occupatonal Task Analyss Program (NOTAP) Navy Personnel Program Support Actvty Buldng 1304, Bolng AFB Washngton, DC 20336 Offce of Cvlan Manpower Management Code 263 Washngton, DC 20390 Chef of Naval Reserve Code 3055 New Orleans, LA 70146 Chef of Naval Operatons OP-987P7 Washngton, DC 20350 ATTN: CAPT H.J.M. Connery Drector Tranng Analyss & Evaluaton Group Code N-OOt Department of the Navy Orlando, FL 32813 ATTN: Dr. Alfred F. Smode LCDR C.F. Logan, USN F-14 Management System COMFTAEWWNGPAC NAS Mramar, CA 92145 Navy Personnel Research and Development Center Code 01 San Dego, CA 92152 Navy Personnel Research and Development Center Code 02 San Dego, CA 92152 ATTN: A.A. Sjoholm Navy Personnel Research and Development Center Code 306 San Dego, CA 92152 ATTN: Dr. J.H. Stenemann
....: :...,.. ;.... V.,:. - T^,.'-,. v.>:,y,.,:' :.....,;,,....,.: s,..v-,; V..V''.r/'/.. ^..:',,^.v..;v;:v,^..:->;/...'v.^: ;.^,.:,^.-..V,. :;.^^: ;.v,^.. ;. :.. ' :.. -^.-y ^. - - '""'" '- --'' - Navy Personnel Research and Development Center San Dego, CA 92152 ATTN: Lbrary Navy Personnel Research and Development Center Code 9041 San Dego, CA 92152 ATTN: Dr. J.D. Fletcher D.M. Gragg, CAPT, MC, USN Head, Educatonal Programs Development Department Naval Health Scences Educaton and Tranng Command Bethesda, MD 20014 STOAC Battelle Memoral nsttute 505 Kng Avenue Columbus, OH 43201 Attenton: H8-1 Code 79986 Drector Naval Research Laboratory Code 2627 Washngton, DC 20390 Pane m Defense Documentaton Center Cameron Staton, Buldng 5 5010 Duke Street Alexandra, VA 22314 Charman Behavoral Scence Department Naval Command and Management Dvson U.S. Naval Academy Luce Hall Annapols, MD 21402 m Chef of Naval Techncal Tranng Naval Ar Staton Memphs (75) Mllngton, TN 38054 ATTN: Dr. Norman J. Kerr Ü w,. summ
: fewaß&fc,,,._..,.,,...,.,,,,..,..,,..,..._,.,,.,,^:,,.,,:...,,:,..,, Pape 4 Chef of Naval Tranng Naval Ar Staton Pensacola, FL 32508 ATTN: CAPT Bruce Stone, USN TACTEC Battelle Memoral nsttute 505 Kng Avenue Columbus, OH 43201 Attenton H8-1 Code 79986 Defense Contract Admnstraton Servces Regon 666 Summer Street Boston, MA 02210 Attenton: DODAAD Code 52202A Command Offcer U.S. Naval Amphbous School Coronado, CA 92155 Dr. James J. Regan Techncal Drector Navy Personnel Research and Development Center San Dego, CA 92152 Chef Bureau of Medcne and Surgery Code 413 Washngton, DC 20372 Mr. Arnold Rubnsten Naval Materal Command (0344) Room 1044, Crystal Plaza 5 Washngton, DC 20360 Superntendent Naval Postgraduate School Monterey, CA 93940 ATTN: Lbrary (Code 2124) Chef of Naval Tranng Support Code N-21, Buldng 45 Naval Ar Staton Pensacola, FL 32508 - * ÄJ^P^^'-^^^V^K^MMaBaÄ HLjHWnmM! «ss&tttwvwktmwww^
:'-\:- ^y^^j-t^ --::^A^^^:~:\:' y. ::Ljt- : -:-b '- ;'. ;:- '.-!.. ;! ^^- VJ^'^VV.-^'.-^-':^ y Dr. H. Wa ace Snako c/o Offce of Naval Research (Code 450) Psychologcal Scences Dvson Arlngton, VA 22217 Armed Forces Staff College Norfolk, VA dszu ATTN: Lbrary Mr. George N, Grane Naval Shp Systems Command SEA 047C12 Department of the Navy Washngton, DC 20362 Commandng Offcer Servce School Command U.S. Naval Tranng Center San Dego, CA 92133 ATTN: Code 3030 Dr. Wllam L. Maloy Prncpal Cvlan Advsor for Educaton and Tranng Naval Tranng Command, Code 00A Pensacola, FL 32508 HQ USAREUR & 7th Army 0DCS0PS USAREUR Drector of GED APO New York 09^03 AR Feld Unt - Leavenworth Post Offce Box 3122 Fort Leavenworth, KS 66027 Dr. Mlton S. Katz, Chef ndvdual Tranng and Performance Evaluaton U.S. Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA 22209 Techncal Drector U.S. Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA 22209 «^ ^^^v^w'w.^;; ^ Page U:. rj5jwy- o -'* ff, ' ÄM,M ' *" w '"'
P^Spy^SSa^f^,.,;; j.^.vr..,:^::;:^^--;^.«.*:'^^,-^-;:^-,^ > V ;^"-; ; '-^,'; j:,-;-r,;» ': ;-:''.' -o.wl Pape 6 Dr. Frank Harrs U.S, Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA 22209 Dr. Joseph Ward U.S. Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA 22209 Mr. James Baker U.S. Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA 22209 Commandant Unted States Array nfantry School ATTN: ATSH - DET Fort Bennng, GA 31905 U.S. Army Research nsttute Commonwealth Buldng Room 239 1300 Wlson Boulevard Arlngton, VA 22209 ATTN: Dr. R. Dusek Dr. Stanley L. Cohen Work Unt Area Leader Organzatonal Development Work Unt Army Research nsttute for the Behavoral and Socal Scences 1300 Wlson Boulevard Arlngton, VA 22209 Dr. Leon H. Nawrock U.S. Array Research nsttute Rosslyn Commonwealth Buldng 1300 Wlson Boulevard Arlngton, VA 22209 Dr. Martn Rockway Techncal Tranng Dvson Lowry Ar Force Base Denver, CO 80230 ; : )
'. :',. E E - Research Branch AF/DPMYAR Randolph AFB, TX 78148 AFHRL/DOJN Stop 63 Lackland AFB, TX 78236. -..--,...,..,,.,? «g p Pape 7 m ä m,$t; 1. 4 Dr. Alfred R. Fregly Ar Force Offce of Scentfc Research/PM Bolng Ar Force Base Washngton, DC 20032 Dapt. Jack Thorpe, USAF Flyng Tranng Dvson AFHRL/FT Wllams AFB, AZ 85224 AFHRL/PED Stop 63 Lackand AFB, TX 78236 nstructonal Technology Branch AF Human Resources Laboratory Lowry AFB, CO 80230 AFHRL/AS (Dr. O.A. Eckstrand Wrght-Patterson Ar Force Base OH 45433 AFHRL (AST/Dr, Ross L. Morgan Wrght Patterson Ar Force Base) OH 45433 Headquarters Electronc Systems Dvson ATTN: Dr. Sylva R. Mayer/MCT LG Hanscom Feld Bedford, MA 01730 Drector, Offce of Manpower Utlzaton Headquarters, Marne Corps (Code MPU) MCB (Buldng 2009) Quatco, VA 22134 r^ Chef, Academc Department Educaton Center Marne Corps Development and Educaton Command Marne Corps Base Quatco, VA 22134 :^_,-^^;2^;_:^.v : :-.:.-..yr.:-:-y:h-^:.^:^ l^; JJ._... : : ^_ - ; ; '
Pape 8 Mr. E.A. Dover 2711 f-cuth Vetch Street Arlngton, VA 22206 Dr. A.K. Slafkosky Scentfc Advsor (Code RD-1) Commandant of the Marne Corps Washngton, DC 20380 Mr. Joseph J. Cowan, Chef Psychologcal Research Branch (P-1) U.S. Coast Guard Headquarters 400 Seventh Street, SW Washngton, DC 20590 Dr. John Ford, Jr. Navy Personnel Research and Development Center Code 304 San Dego, CA 92152 LCDR Charles Thesen, Jr., MSC, USN, 4024 Naval Ar Development Center Warmnster, PA 18974 Mltary Assstant for Human Resources Offce of the Secretary of Defense Room 3D129, Pentagon Washngton, DC 203 v " Advanced Research Projects Agency Admnstratve Servces 1400 Wlson Blvd Arlngton, VA 22209 ATTN: Ardella Holloway Dr. Harold F. O'Nel, Jr. Advanced Research Projects Agency Human Resources Research Offce 1400 Wlson Boulevard Arlngton, VA 22209 Dr. Robert Young Advanced Research Projects Agency Human Resources Research Offce 1400 Wlson Blvd Arlngton, VA 22209
mmwmmmww?w ^^ ' "' " '< - ^' ^[ [^^ : :-'--- jr: -''' ', -: -'-.-:-..,..-...,. -:.-:...., - --. Ov [3- Pape 9 v;- :; ä : : Ü 1 f 1 Dr. Wllam Gorham, Drector Personnel Research and Development Center U.S. Cvl Servce Commsson 1900 E Street, N.W. Washngton, DC 20^15 Dr. Marshall S. Smth Assstant Actng drector Program on Essental Sklls Natonal nsttute of Educaton Brown Buldng, Room 815 19th and M Streets, N.W. Washngton, DC 20208 Dr. Carl Frederksen Learnng Dvson, Basc S':lls Group Natonal nsttute of Educaton 1200 19th Street, N.W. Washnton, DC 20208 Dr. Erc McWllams Program Manager Technology and Systems, TE Natonal Scence Foundaton Washngton, DC 20550 Dr. Scarva Anderson Executve Drector for Specal Development Educatonal Testng Servce Prnceton, NJ 08540 m Professor Keth Wescourt Stanford Unversty nst. for Mathematcal Studes n the Socal Scences Stanford, CA 94305 Dr. Bernard M. Bass Unversty of Rochester Management Research Center Rochester, NY 14627 Century Research Corporaton 4113 Lee Hghway Arlngton, VA 22207 m Dr. Robert Glaser, Drector Unversty of Pttsburgh Learnng Research and Development Center Pttsburgh, PA 15213 SS^SS^S^^T^sss^^ 11 MMmsnMMHRM JüKKBK'f."»vt Mh..- o :,'.-. ^ - -. -.-V,.'....,.--.. (.. -:'-: (.'' '
;- ;,:,^gmaaay»bmg'a»^5w^ Page 10 Dr. Kenneth E. Clark Unversty of Rochester College of Arts and Scences Rver Campus Staton Rochester, NY 14627 ERC Processng and Reference Faclty 4833 Rugby Avenue Bethesda, MD 20014 Dr. Vctor Felds Department of Psychology Montgomery College Rockvlle, MD 20850 Dr. Henry J. Hamburger Unversty of Calforna School of Socal Scences rvne, CA 92664 Dr. Rchard S. Hatch Decson Systems Assocates nc 11428 Rockvlle Pke Rockvlle, MD 20852 Dr. M.D. Havron Human Scences Research, nc. Westgate ndustral Park 7710'01d Sprnghouse Road McLean, VA 22101 Dr. Lawrence B. Johnson Larence Johnson and Assocates, nc. 200 S. Street, NW, Sute 502 Washngton, DC 20009 '1 V Dr. Davd Klahr Carnege-Mellon Unversty Graduate School of ndustral Admn. Pttsburgh, PA 15213 Dr. Robert R. Macke Human Factors Research, nc 6780 Cortona Drve Santa Barbara Research Park Goleta, CA 93017 m: :_ ssss;
:............,.....,..,,... Dr. Andrew R. Molnar Technolopcal nnovatons n Educaton Natonal Scence Foundaton Washngton, DC 20550 Dr. Leo Munday Vce Presdent Amercan College Testng Program P.O. Box 168 owa- Cty, A 52240 Dr. Donald A. Norman Unversty of Calforna, San Dego Center for Human nformaton Processng La Jolla, CA 92037 Pare 11 Mr. Lug Petrullo 2431 North Edgewood Street Arlngton, VA 22207 Dr. Dane M. Ramsey-Klee R-K Research & System Desgn 3947 Rdgemont Drve Malbu, CA 90265 Dr. Joseph W. Rgney Behavoral Technology Laboratores Unversty of Southern Calforna 3717 South Gr-c Los Angeles, CA 90007 Dr. Leonard L. Rosenbaum, Charman Department of Psychology Montgomery college Rockvlle, MD 20850 Dr. George E. Rowland Rowland and Company, nc. P.O. Box 61 Haddonfeld, NJ 08033 Dr. Arthur. Segel Appled Psychologcal Servces and Scence Center 404 East Lancaster Avenue Wayne, PA 19087 1 71
.,. - - : -;. :-:... -. --- -[-.: :- ;. --... :. :. : : Pape 12 Dr. Benton J. Underwood Northv^estern Unversty Department of PsycholoRy Evanston, L 60201 Dr. Davd J. Wess Unversty of Mnnesota Department of Psychology Mnneapols, MN 55455 Dr. Kenneth Wexler Unversty of Calforna School of Socal Scences rvne, CA 92664 Dr. Anta West Denver Research nsttute Unversty of Denver Denver, CO 80210 Dr. Rchard Atknson Natonal Scence Foundaton Washngton, D.C. Professor Rchard D. Snow Department of Educatonal Psychology Stanford Unversty Stanford, Calf. 94305 John H. Wolfe U.S. Navy Personnel Research & Development Center San Dego, Calf. 92152 Dr. Wllam Mann nformaton Scences nsttute 4676 Admralty Way Marna del Ray, Calf. 90291 Professor Derek Sleeman Dept. of Computer Scence The Unversty Leeds 2 England Dr. John Annett Deparvment of Psychology The Unversty of Warwck Coventry CV47AL ENGLAND -r.-k : :: '- < :- - ^r''' - '- -
'.; Page 13 «Mr. Samuel Ball Educatonal Testng Servce Prnceton, NJ 08540 ü ff 11 u Dr. Gerald V. Barrett Unversty of Akron Department of Psychology Akron, OH 44325 Dr. Ronald P. Carver School of Educaton Unversty of Mssour-Kansas Cty 5100 Rockhll Road Kansas Cty, MO 64110 Dr. Ruth Day Yale Unversty Department of Psychology 2 Hllhouse Avenue New Haven, CT 06520 Dr. Edwn A, Fleshman Vstng Professor Unversty of Calforna Graduate School of Admnstraton rvne, Calforna 92664 HumRRO Central Dvson 400 Plaza Buldng Pace Boulevard at Farfeld Drve Pensacola, FL 32505 HumRRO/Western Dvson 27857 Berwck Drve Carmel, CA 93921 ATTN: Lbrary HumRRO Central Dvson/Columbus Offce Sute 23, 2601 Cross Country dr,e Columbus, GA 31906 HumRRO/Western Dvson 27857 Berwck Drve Carmel, CA 93921 ATTN: Dr. Robert Vneberg HumRRO Joseph A. Austn Buldng 1939 Goldsmth Lane Lousvlle, KY 40218 -l»ssg^^. :.,- s. '-^
l:'m;.->!s!&mt*vw*^mmmzw! ^^,.,.^>v,, ;,^-... - -,..:.;... ; :;-, ^, T,;,-. /,.^r^:j/ v,, ; ;! ;. B.>-yv'v--^-^ ^ ^l^- - ^-V.v- Pape 14 Dr. Arnold F. Kanarck Honeywell, nc. 2600 Rdge Parkway Mnneapols, MN 55413 Dr. Roger A. Kaufman U.S. nternatonal Unversty Graduate School of Human Behavor Ellott Campus 8655 E. Pomerada Road San Dego, CA 92124 Dr. Steven W. Keele Unversty of Oregon Department of Psychology Eugene, OR 97403 Dr. Alma E. Lantz Unversty of Denver Denver Research nsttute ndustral Economcs Dvson Denver, CO 80210 Mr. Bran McNally Educatonal Testng Servce Prnceton, NJ 08540 Mr. A.J. Pesch, Presdent Eclectech Assocates, nc. P.O. Box 178 North Stonngton, CT 06359 Dr. Steven M. Pne Unversty of Mnnesota Department of Psychology Mnneapols, MN 55455 M H Mr. Denns J. Sullvan c/o HASC, Buldng 119, M, P.O. Box 90515 Los Angeles, CA 90009 Dr. Patrck Suppes Stanford Unversty nsttute for Mathematcal Studes n the Socal Scences Stanford, CA 94305 '"".;-;^^':v;vv\-;-.v--^--'- : J'^='''-"^>;'' : -'^^33TVf^^tL-v'::' ^^y* wamjräsraw «SKWEWTD-M -^^rr >. v.:-.-..v.m.vjt- -- ^p^^mt^h^*^^
-.... ' --; Pa/re 15 Dr. K.W. Uncapher Unversty of Southern Calforna nformaton Scences nsttute ^676 Admralty Way Marna Del Rey, CA 90291 Dr. Carl R. Vest Battelle Memoral nsttute Washngton Operatons 2030 M Street, N.W. Washngton, DC 20036 S Dr. John J. Collns Vce Presdent Essex Corporaton 6305 Camnto Estrellado San Dego, CA 92120 Mr. Charles R. Rupp Advanced W/C Development Engneerng General Electrc Company 100 Plastcs Avenue Pttsfeld, MA 01201 f m m m * 1 E