Proposal for a CG Educational Content Online Submission and Reviewing System Sónia A. Assunção LEIC, IST saa@virtual.inesc.pt Frederico C. Figueiredo LEIC, IST fepf@virtual.inesc.pt Joaquim A. Jorge INESC/DEI/IST jorgej@acm.org ABSTRACT Interactive Computer Graphics has evolved considerably over the past few decades. As computer science has continued to evolve and gain new substance, educators have come to master new content and achieve deeper understandings of the nature of computers and imagery. As the discipline becomes more mature, the need for high-quality resources and the comparative ease with which these materials can be generated suggests that an important service to the community of educators could lie in offering high-quality educational materials so that both young and seasoned educators alike can both benefit from and contribute to the work of others to achieve a higher standard of teaching worldwide. The purpose of our work is to provide tools to foster such a Community of Computer Graphics educators, by providing the means for their work to be appraised, assessed and made available to others, through an online server for refereed educational content in Interactive Computer Graphics. In this paper we describe the basis and highlight some of the staring requirements for an Online Educational Module Submission and Reviewing System and Educational Resources service using a webbased groupware application that supports the review process for technical contributions to conferences. INTRODUCTION Supporting online communities is somewhat of a difficult task given the many tools that are required to provide a sense of belonging. In a recent proposal, Cunningham and Jorge [1] identify four main functions that a system geared towards educators should provide: a) A process for refereeing content submitted from individuals or groups in our field, including content now available at some online sites. b) A structure for online content to make it straightforward for educators to find materials for whatever courses they are teaching. c) Mechanisms to make members of the community know which resources are and which are not available, to encourage them to develop materials in areas in which such content is lacking. And finally d) tools to support (volunteer) people in managing the submission and refereeing process, to make the whole effort self-sustainable on a reasonable budget both in financial and human resources. The present paper describes a first attempt at building a prototype system to support these functions in a simple and sustainable manner. THE SYSTEM Web-based tools for publishing and managing academic material are becoming easier to use, and widely available. Currently, there are many resources on the Internet although much of the material is not clearly documented or trivially adaptable. Further many packages require modifications and additional work to make them usable in straightforward manner. Our main goal is to develop an online resource content in computer graphics education that can be used by volunteers with minimal maintenance effort. In this way, it will help CG content providers and also provide a foundation and structure for the online content and ways to let members of the computer graphics community know what resources are and are not available. Our aim is to support all phases of the submission and review process, including preparation of the courseware material in a format suitable to be downloadable and immediately usable by educators. Support for the Refereeing Process We aim at adapting one of the major conference refereeing systems available such as Cyber- Chair [2] or similar systems, towards meeting or substantially promoting these goals. This system already supports work in two different areas: a private area accessible to server maintainers, editorial board members and reviewers and a public area available to submitters and community members at large.
Figure 1: Overall Workflow overview The Private area includes support for all the invis i- ble editorial and managerial processes that include the editorial section, the reviewer s reminders. The Public area offers didactic contents search and a publication system, including sections for curriculum structure and courseware modules to be made available. ADMINISTRATIVE TASKS There are many tasks that a refereeing system needs to support. Among them we can identify Submitting course materials, including interactive content, grading mo dules, movies, images, textbook-like materials, etc. Once these materials have been submitted they should go through a preliminary screening, before being assigned to reviewers. The system should also support emailing reminders for reviewers. Referees need to submit their reviews in a timely fashion. When discussions arise there should be support for reviewer comment threads. Another important function is sending letters to authors and possibly supports different tracks/curricula for submissions (abstracts/course outlines, text book materials in PDF format, interactive mo dules, book reviews, short letters, among others).
The system uses the web for submission and distribution of all data that eventually will result in courseware. The authors use web forms to submit modules and other information. The reviewers use the web to download materials and to submit, update, discuss or read reviews. The Editor-in-chief is able to follow the flow of information, submission and refereeing work through the web, using several overviews that are timely generated. In this way, all people involved work towards the online curricula in a collaborative way. Figure 1 illustrates the overall workflow detailing the tasks involved in the reviewing system. Resource Maintenance Another important role for the server is to make available in a timely and update manner a set of common resources for educators, such as book Figure 2: Workflow for authors to have a prototype supporting most of the refereeing and submission functions available in time for the workshop. MODULE REVIEW PROCESS In what follows we describe briefly the flow work and information involving three major entities in the process of submitting, reviewing and publis h- ing educational content. For the purposes of this discussion, we assume that the server is managed by one (or more) editors-in-chief, assisted by an editorial board composed of and/or assisted by reviewers. Generally, the reviewing process starts by authors submitting modules for publication in the refereed server. The Editor in Chief (EIC) checks modules against a set of minimum requisites and selects for reviewing those mo dules satisfying a minimum of acceptability criteria, e.g. subject and scope. Modules thus selected, are then reviews, curricula available in different universities throughout the world and national programs. The system to develop should provide minimal support for editorial content of these different sections involving minimal effort from volunteer section editors. The tasks supported should include as a minimum mo dules for submitting content, to notify editors of the arrival of new materials and to publish these with minimal formatting effort. We plan assigned to at least three reviewers selected by the EIC, according to their declared preferences and skills. After reviewers have produced their analysis of content, the EIC decides whether a submission is accepted, whether must be revised according to comments from reviewers, or whether it will not be accepted. For accepted modules, the maintainer gets sent the final version for publishing. We will now present in some detail the main tasks to be
Figure 3: Workflow for Reviewers performed by Authors, EIC, Reviewers and Maintainer. Authors When authors submit a module for the first time, they need to fill in an author registration form including their contact information, title of the submission, abstract, keywords and a set of files that comprise the submission. The system then generates a unique identifier (username and password) to be used by authors when accessing the system to check module status, resubmit or otherwise interacting with the editorial board concerning this submission. This process is illustrated in Figure 2. After first submission the modules get sent to the EIC, who can reject the submission on formal grounds as seen above, which terminates the process. For modules accepted for reviewing by the Editorial Board, these can be either accept without major changes after reviewing or the module may get sent back to authors with substantial comments from reviewers for reformulation and resubmission. During the reviewing process, authors can check the status of their submissions to follow the progress of reviewing. A submitted module can be in one of several states during the review process. Accordingly, authors can check whether their submission has been a) accepted for reviewing, b) sent to reviewers, c) rejected, d) sent back for revision, e) re-submitted, f) accepted for publication or g) published. Authors of educational modules accepted for publication are notified by the EIC of this acceptance. They can still review the module for final submission or check the (anonymous) comments made by referees. If the reviewers feel that substantial changes need to be made, another review cycle is started after the revised module is resubmitted. Reviewers Reviewers are initially designated either by the EIC or one of the other members of the Editorial Board. In the latter case, they will have access to a reviewer s registration form (to be implemented in a future version) to indicate their profiles, which describe both interests and expertise. This process is illustrated in Figure 3. The reviewers profile is used by the EIC to assign and distribute the submitted mo dules (a process designated by binding). Reviewers can also indicate their preferences regarding which mo dules they would like to review. In order to support both the EIC and reviewers on the binding process the system provides them with dynamically generated web pages containing overviews of submitted modules. After modules are distributed, the system generates a personal (password protected) web page for each reviewer. Using their pages reviewers are able to: (1) download the assigned modules, submit comments and reviews, check the state of the reviews of each assigned module (this changes each time a review is submitted by any of the other reviewers) and (4) consult the comments made by other reviewers.
Figure 4: Workflow for Editor-in-Chief A reviewer can only check the comments made by others for a selected module after they have submitted their own review. In order to solve conflicts arising out of conflicting evaluations reviewers can communicate among them selves in a straightforward manner. Modules are classified by the reviewers using the following classification: a) Advocate/Accept; b) Accept, but could be reject); c) Reject, but could be accept; d) Reject. Editor in Chief The Editor-in-Chief (EIC) is responsible for managing the submission and the review process. The EIC workflow is described in Figure 4. The EIC checks all submitted modules against a set of minimal requirements (as described earlier in the Section on Review Process) and is able to reject modules that fail on formal grounds, terminating the review process. Modules thus accepted are assigned to reviewers according with their expertise and preferences (each module is assigned to at least three reviewers). After assigning the modules to reviewers the main task of the EIC is to mo nitor the review process by using several overview screens, which are generated at regular intervals. Checking the status on all reviews enables the EIC to send reminders to reviewers who are late in submitting their evaluations. In extreme cases the EIC can assign the selected module to other reviewers. After all reviewers have produced their analysis of content for a selected module, the EIC checks and resolves any existent conflicts and decides whether a submission is (a) accepted, whether (b) must be revised according to comments from reviewers, or whether it will (c) not be accepted. The EIC sends an email with feedback to authors. For accepted modules, the EIC sends the final version of the module to the maintainer, for publishing when authors finally submit it. Maintainer The maintainer (who could be the same person as the EIC) receives the final version of the modules that were accepted for publishing. His or her role is then to prepare and publish this as an accepted contribution to the refereed server. This may involve not only some extra formalisms, but on a more important, to catalog, classify accepted contributions so that they can be retrieved at a letter time. This task is very important and directly affects the usefulness of the reefed server. This we decided to isolate and assign this role to a distinct entity who might or might not be performed by (one of) the editor(s) in chief. CONCLUSIONS We have presented an overview and high-level description of the workflow to support a refereed content server for CG educational materials. Even though we focus on refereed content this does not imply that the server could also host non-refereed information. We feel that the added value of such a server is directly related to the stringency and rigor of the refereeing process. This justifies in our view the decision to support such a process. REFERENCES [1] A proposal to develop refereed online content, Steve Cunningham and Joaquim Jorge, October 2001. [2] CyberChair: A Web-Based Groupware Application to Facilitate the Paper Reviewing Process