index  

Summary

Digital Insight description

Digital Insight Partners

Digital Insight Technologies



References

Contact

Researchers, educators, and classroom teachers are increasingly relying on video data to build case histories, document instructional practices, track cognitive development, and develop new curricula. The use of video and other related data - images of student work, transcripts of dialogue, test scores, observational notes, and the like - opens up new areas of inquiry and gives us new ways of investigating traditional questions. For example, long-standing questions about conceptual change and classroom discourse are sustained by the capacity to archive carefully constructed video records over months and even years of classroom learning (e.g., Jordan & Henderson, 1995; Strom, Kemeny, Lehrer & Forman, 2001) However, the expanding role of video and other accompanying multimedia data also presents new challenges in data acquisition, management, analysis, and dissemination. The Digital Insight project promises to automate elements of video data analysis that are currently "handcrafted." Because handcrafting takes hundreds of hours, progress in the field is painfully slow. Moreover, because each project that relies on video data must "reinvent the wheel," it is almost impossible for even closely related research teams to learn from each other since their tools, and their associated methods, are often incommensurate. The research and development proposed here will provide a level of consistency in method and process that will have a profound impact on both research and instruction in math and science.

The Wisconsin Center for Education Research (WCER), the TalkBank Project, and the Data Intensive Computing Environments group at the San Diego Supercomputer Center (SDSC) have formed a partnership to address the challenges of video-based data collection, analysis, and dissemination. Called Digital Insight, the project will both build capacity for analyzing rich, complex data sets and building histories of student learning for use in research and instruction in math and science education. We will, at the same time, study the dynamics of successful collaborative teams and their interaction with the tools developed and services provided in an attempt to discover and disseminate more efficient practices and more powerful analytical tools. We have both video-analysis tools and a research-based online workflow environment ready to support participating research teams. We also have a prototype process model for supporting collaborative research. Digital Insight developers and participating research teams will use and augment our tools and help us develop and refine our process model as they create new research-based materials for high-impact professional development, teacher educator training, and instruction. Our research is aimed squarely at the intersection of this effort to build capacity and to understand the consequences of this new capacity for research and professional development.

Digital Insight will provide participating research teams with access to large-scale online storage at WCER with an integrated multimedia catalog system based on the recently released Metadata Encoding & Transmission Standard (http://www.loc.gov/standards/mets/). SDSC will provide an additional 20 terabytes of tape and disk storage for collection management. The scale of this commitment will allow projects to bring extremely large collections online, a feat that would be difficult for any individual research team to accomplish. In addition, developers at WCER and TalkBank are working together on a small start-up project to build open-source, cross-platform tools for multiuser transcription, annotation, dissemination, and analysis of complete collections of video with related student data.

The proposed Digital Insight project has the potential to produce a major leap forward in the study of science, technology, engineering, and mathematics (STEM) education by developing improved video interface technologies, research methodology, human subject protection, data management, and mixed methods analysis, and by enabling the use of archived multimedia data for secondary analysis and teaching. Digital Insight will link the existing tools (and those to be developed) into a research infrastructure for the investigation and design of learning-in-context. Digital Insight will allow its partners to make new arguments and bring more forms of data to bear on a problem. It will also facilitate the full participation of collaborators at other institutions. Digital Insightís impact will be felt across a wide range of currently funded research and will generate important new resources for professional development and classroom curriculum.

The overarching goals of the Digital Insight project are to:

  1. Provide access to new tools for research and dissemination to projects engaged in STEM teacher education and professional development;
  2. Develop and study a prototype research collection management environment that enables collaboration at a distance;
  3. Implement and test our AMAD[1] process model;
  4. Develop and embed robust human subject protection procedures and collection management heuristics that will encourage and support new data use models and ensure that rights for data use are associated with the data itself within the catalog system;
  5. Follow a user-centered, participatory design process that encourages researcher engagement in developing and refining the features of the system; and
  6. Evaluate the effectiveness of Digital Insight as a model for collaborative research and development.

The ubiquity and instructional power of video in teacher professional development, studies of classroom practice, and preservice teacher education ensure a broad target audience for the Digital Insight tools and processes.

Range of Use Models

Use models for Digital Insight range along a number of dimensions. Three of the most important dimensions are the granularity of the analysis, the temporal resolution of the driving research questions, and the size of the data set to be analyzed. Potential uses of the Digital Insight tools and processes range from fine-grained studies of human interaction (e.g., taking turns in conversation) to large, ethnographic studies of change over years of instruction. One particularly relevant example is a special edition of The Journal of the Learning Sciences (Sfard & McClain eds., 2002) in which six different scholars apply their particular theoretical approaches to the same video of seventh gradersí being introduced to statistical concepts. Brian MacWhinney extends this work by providing an example of what he calls collaborative analysis,[2] in which he aggregates the analytical markup of the six authors to give a combined, comparative view of the various methodical approaches.

Other researchers use video in traditional A-B comparisons of various treatments. This approach usually takes the form of studies of control and treatment classroom video that covers the same instructional area but contrasts different pedagogical techniques or access to different resources. The Third International Mathematics and Science Study-Repeat (TIMSS-R) study falls into the larger end of such mid-range studies. Rather than a controlled experiment, however, TIMSS-R provides a quasi-experiment by examining variation within and between classrooms and within and between participating countries.† Other large-scale studies, such as Rich Lehrerís work on building long-term histories of individual studentsí cognitive development across a number of years, take quite a different approach. Lehrerís research combines both fine-grained analysis of case study studentsí speech and numerous historical accounts of the studentsí work. Lehrer then compares and contrasts these individual longitudinal studies to explore the impact of both natural variance and variance in treatment.

Digital Insightís tools will be relevant across this broad range of approaches - from individual researchers and teams engaged in collaborative analysis to long-term studies that span many sites and cross institutional boundaries.

Use and Enhancement of Existing Technologies

The Digital Insight project will build on the work of SDSC to extend the benefits of its innovative analytical tools to education researchers. The secure data storage and management environment, called the Storage Resource Broker (Rajasekar, Wan, & Moore, 2002; Rajasekar & Wan, 2002; Rajasekar & Moore, 2001) and developed at SDSC will provide Internet-based access to large, complex collections of research video and other related evidence. The infrastructure being developed by SDSC reflects a rich understanding of cross-disciplinary research and represents a great capacity for helping organizations understand Ďhow they manage what they know.í (Davenport & Prusak, 1998). Currently, the most ambitious project to develop a standard for managing, exploring, and creating links within heterogeneous collections of multimedia data is being conducted by the Library of Congress and its partners within the Digital Library Federation (http://www.diglib.org/). The Library of Congress is already a collaborator with SDSC on a prototype project to operationalize distributed collection management using the SDSC-based storage grid. We will take advantage of the work that has already been done on federating and searching complex collections.

The TalkBank development team will work with WCER programmers on the creation of open-source tools for video transcription, linkage, and analysis that follow from the work they have done in the area of analysis of child language. CLAN, their transcript analysis tool, has been an important tool for linguistic analysis across the globe.[3] In particular, the markup language CHAT that grew out of the use of CLAN has become a dominant form of linguistic markup and is supported by many of the other analytical tools available for linguistic research. The TalkBank teamís experience in developing and supporting analytical software for research in the social sciences will provide a vital backdrop as we move towards a more complex use environment and model.

Transana's Main screen:

WCER brings to the table a tool for video analysis - Transana.[4] Transana is available for both single users and research teams and is under active development. Transana is designed to facilitate the transcription and analysis of video data. It provides a way to view video, create a transcript, and link places in the transcript to frames in the video. It provides tools for identifying and organizing analytically interesting portions of videos, as well as for attaching keywords to those video clips. It also features database and file manipulation tools that facilitate the organization and storage of large collections of digitized video.

Transana consists of a Sound Waveform Window, a Video Window, a Transcript Window, and a Database Tree. The Sound Waveform presents a visual representation of the audio track of the video being analyzed, allowing great precision in selecting portions of the video for analysis. The Transcript Window is a word-processing window that allows users to easily create and edit transcripts for videos. A frame in the video can be linked to a position in the transcript by inserting a time code. Such links enable Transana to synchronize video playback with portions of the transcript. The Database Tree gives a visual representation of the organization of data within Transana, making it easy to manage even large collections of analytically interesting video clips. Analytic keywords can be assigned to segments of video data, and Transanaís search features allow for increasingly sophisticated data mining. One can then group video clips to assist in the analysis of data.

Existing Digital Insight tools, notably CLAN and Transana, are and will remain available for free download to anyone interested in the software. All improvements to this software and any additional software developed will be made available as well - both in executable form and as source code for those users who would like to make their own extensions to the work. The only resource that will not be available to research teams that are not active participants in the Digital Insight project is access to the projectís large-scale storage systems. However, one of the projectís goals is to provide a detailed model for building local storage resources and integrating them into a local Digital Insight-like infrastructure. We also intend to extend the SDSC-based storage management system (Storage Resource Broker [SRB]; http://www.npaci.edu/DICE/SRB) to other research institutions. These relationships will be negotiated on a case-by-case basis since they require a level of trust and interoperation that exceeds that necessary for using the stand-alone tools. In the meantime, the multiuser version of Transana is already capable of supporting multisite collaboration. What it lacks (and what Digital Insight will add) is the ability to support more sophisticated collection management and search technologies that are available within the SRB.

Methodology

Digital Insight will research the applicability of new technologies to a wide spectrum of education research projects. Our selection of research efforts for study (see pp. 11-14) is intended to sample from this spectrum. Accordingly, we will support the research teamsí work through the AMAD process model as they transform their multimedia data into scholarly and instructional deliverables. Our primary technical focus will be to identify the operational characteristics of the AMAD process and implement best-practice strategies in collaboration with the research teams.

The research design for this study is that of the quasi-experiment with comparative cases. We will establish baseline characteristics of both control groups (teams using analog case-building methods) and treatment groups (participating research teams using digital case-building methods) at the beginning of the project. The treatment will vary according to which technologies are used in the acquisition, management, analysis, and dissemination of research objects.[5] The comparative case study approach we will use is based on the designs suggested by Barlow, Hayes, and Nelson-Gray (1998) and Kazdin (1998) in their excellent outlines for clinical studies of comparative cases involving scientist-practitioners. We will also take into account the work of Gummesson (2000) as we examine the interaction between academic researchers and consultants in qualitative case studies. Collaboration between technical and substantive experts will be an important aspect of the interaction between the programming and research support consultants on the core development team and the participating research teams.

We will create case studies for each of the participating research teams described below (pp. 11-14). The project staff includes researchers with experience in ethnography, quantitative analysis, formative and summative evaluation, workflow analysis, interface design, and information system design. Data collection efforts will include interviews with PIs, direct observation of research team meetings, and documentation and analysis of the evolution of research teamsí analytical models as they incorporate Digital Insight tools into their work. We will also document the interaction between the research teams and the development team to study how best to coordinate the work between researchers and those building research tools.

The PI, Christopher Thorn, will lead and coordinate the ethnographic study of the impact of Digital Insight on the scholarly processes and outputs of the participating research teams. Sarah Mason, the project manager, will lead the evaluation efforts and coordinate the work of the library and information systems staff who will provide assistance with interface and information system design. David Woods will lead the software development efforts. His unusual background in counseling psychology and analytical software design will be an asset as he provides both software design and research support to participating research teams. Woods will also support the ethnographic study by conducting research staff interviews and using detailed participant protocols that require the research teams to reflect on their own work and the impact of Digital Insight tools and process on their work. The Digital Insight project team will provide ongoing feedback to the participating research teams about our analysis of their work and the impact of our collaboration. We will also provide a framework for cross-case comparisons that will contrast the structural and behavioral similarities and differences between the teams. These comparisons will provide the context for analyzing the scholarly outcomes across the cases. Our goal is to be able to describe several best-path models for the efficient use of video data in research and teaching.

Digital Insight Process Model (AMAD)

A four-phase system of consecutive operations can describe the overall process of most education research activities. These four operations consist of the acquisition, management, analysis, and dissemination of findings based on data (AMAD). This general model can be adapted to any project.

The importance of the AMAD model lies partly in two of its benefits: it emphasizes phases of the research process that might otherwise receive insufficient attention, and it treats the various operations of education research as part of an integrated system of knowledge production, thus allowing for synergistic improvements in the research process at the system level. For example, the AMAD model will help to clarify how acquiring data in a particular format affects not only its management and analysis, but also the practical range of possible products for dissemination. The usefulness of the model is due both to its inherent applicability to a wide range of research approaches and to its ability to be customized to fit each approach. The model can accommodate a typical research project that simply acquires, manages, analyzes, and disseminates data, but it can also address the needs of projects that repeat the AMAD sequence or that perform the AMAD operations in nonsequential order (e.g., analyze the data as it is being acquired).

User-Centered Design Strategy and Formative Evaluation

To maximize end-product utility and usability, Digital Insight will coordinate the development of technological tools (video analysis, data management, and presentation software) with the development of user practices within an iterative and user-centered design framework. User-centered design principles help streamline product evolution by providing developers with design recommendations throughout the development cycle, especially during early development efforts when significant changes can be made to design goals with minimal costs (Baecker, Grudin, Buxton, & Greenberg, 1995; Nielsen, 2000; Rouse, 1991; Shneiderman, 1998). Adopting user-centered design practices will also help facilitate and maximize the benefits of interdisciplinary work between development teams and research teams. The initial identification of user groups and assessment of user needs will occur during the first year of the project and will provide baseline information for further design practices (Nielsen, 1994).

We will construct process flow models of video-based education research with the help of task analysis and other direct observation techniques. These process flow models will (a) document the process by which identified users actually go about conducting such research; (b) allow us to target specific tasks as inefficient or otherwise problematic; and (c) allow developers and researchers to calculate metrics that can serve as tools for evaluating various methods. We will also explore the use of appreciative intervention to engage users in speculation about possible future uses of existing tools and possible additional uses based on proposed feature set changes (Karasti, 2001).

Research and Evaluation Questions

1.      How can we improve the use of video in education research, and how will digitizing video and using the AMAD approach enhance the research process?

We will observe the work processes of researchers using traditional analog methods and those using Digital Insight tools to build video case histories. To help us establish a baseline for the differences between these two sets of researchers, we will interview team members about their research goals and their expectations about the workload associated with the research process. In the case of our participating teams, we will also debrief them about recommended refinements to the tools and workflow/process models. The implementation of each round of new tool developments will be followed by an additional wave of field observations and interviews.

2.      How will the use of analytical tools for combining video and transcript analysis - in addition to the inclusion of other types of evidence (images of student work, audio, etc.) - improve the research findings and their application for participating research projects?

Researchers will be asked to act as participant observers and complete regular field logs recording problems encountered, suggestions, new insights, the impact of the tools on the quality or quantity of their research output, and similar observations. Members of the evaluation team will also collect research output - for example, presentations, papers, and articles - and compare them to previous work of the same authors (when available) and to the work being done by control group teams and their peers, as identified by each team. We will also interview research team members to explore the impact of Digital Insight tools on their scholarly work. We expect to see changes in dissemination methods, an increasing sophistication of argumentation, and the inclusion of new forms of evidence in scholarly debate.

3.      What impact will an integrated, secure, Internet-based data management environment have on education research conducted by geographically distributed, interdisciplinary teams?

The increasing importance of supporting distributed teams is borne out by the constitution and collaborative methods of the core team itself. Secure, online collaborative tools will be essential for supporting rapid, efficient software development and effective research workflow processes across geographically dispersed teams. The same field techniques described above will be used to document the coordination of activities both within the control groups and within (and across) the participating teams. We will explore the participating researchersí impressions of (a) their own effectiveness working at a distance and (b) the impact these technologies have on establishing and building team cohesiveness and trust.

4.      What models for linking human subject confidentiality requirements can be embedded within the Digital Insight system?

Federal guidelines for human subject protection and the emerging technical requirements for securing clinical research environments are still in flux. However, Digital Insight will provide an opportunity to field-test the TalkBank model of informed consent for the use of video research data. Teams will be encouraged to use and refine the TalkBank model as they interact with their respective institutional review boards and with their research subjects. Refinements to the release model and to the practices for documenting consent and use rights will be one of the important scholarly outcomes for the participating universities.[6]

5.      Is a collaborative process model derived from basic research collaboratives in manufacturing transferable to education research?

The scope of Digital Insight is broad enough to provide some systematic insights into the dynamics of successful collaborative research teams. Recent trends toward large awards to multi-institution research partnerships by the U.S. Department of Education and the National Science Foundation suggest that both of these agencies view collaborative research efforts as an important source of innovation. We will ask our advisory panel to consider including research teams from several of these large groups - such as the new NSF-sponsored Centers for Learning and Teaching - when it recommends new research teams for inclusion in Years 2 and 3. Data collected to answer Question 3 above will allow to us explore the conditions necessary for successful research collaboration.

6.      What types of users and uses are best supported by these new tools and procedures, and what types of users and uses seem hard to influence?

In interviews and observations over the life of the study, we will be looking for indicators of system failure and research needs that remain unmet. Research teams (including the core team) will be asked to reflect on the barriers to implementation and the unanticipated costs associated with participation in the project. The formative evaluation of the introduction of new tools and processes will be fed back into the design process and will provide information for a summative impact analysis at the end of the project.

Project Foci and Impact

Online resources already developed by the Digital Insight team and related information about our research partners can be found at http://www.wcer.wisc.edu/digitalinsight. Our programming efforts will center on the development of user-friendly front-end tools that can take advantage of the federated storage management environment developed by SDSC.[7] These front-end tools will include open-source tools[8] such as Transana for the analysis of transcripts synchronized to video and audio files. The Digital Insight project will provide education researchers with tools for extracting event-based data from video streams and correlating these data with, for example, field notes, images of student work, lesson plans, and assessment data.

Within the context of an advanced human-computer interface based on tools currently under development, Digital Insight will include features such as code-based causal grouping and event pattern interpretation. Analysts will be able to apply unique coding schemes to portions of the transcripts of a video collection, as well as to portions of the video itself. This ability is vital for supporting an iterative research process that allows for the sort of ongoing or termless approach to inquiry suggested by Koschmann (in press) and Koschmann, Kelson, Feltovich, and Barrows (1996). The resulting analytical environment will provide researchers with tools for theory building and hypothesis testing. This work will take advantage of the research done by designers and critics of groupware technologies. Of particular importance will be the caveats presented by Grudin (1990, 1994) and cited in Choo, Detlor, and Turnbull (2000, pp. 92-93) on the pitfalls of designing without understanding and reflecting the incentive structure of key users.

Digital Insight ROLE and Themes

Digital Insight will support multiple ROLE program themes. First, by providing broad access to large-scale, distributed storage and free, feature-rich, open-source analytical tools, Digital Insight will allow for a rapid scale-up of projects engaged in analysis and manipulation of digital video data focused on student learning. By linking the analytical tools to the existing storage grid development efforts under way at SDSC, Digital Insight will enable participants to take advantage of the newest developments in the NSF-sponsored TeraGrid program. This involvement will also provide SDSC developers with a broad, education-oriented community to take part in its participatory design efforts.

Second, Digital Insightís impact on improving human capacity through education research and instruction will be profound and transformative. The construction of histories of learning is a vital step toward developing and testing new, high-strength instructional approaches. The work of the participating research teams is aimed at supporting early acquisition of foundational skills and improving studentsí transition to increasingly complex science and mathematics learning. In particular, the teams are working on increasing the analytical capacity of teams doing work on cognitive testing and development, early math education, preservice teacher training in math and science, and training of teacher educators.

Finally, Digital Insightís focus on the AMAD process model has the potential to provide new insights into the dynamics of multisite, interdisciplinary research practices. The introduction of a rich collection management architecture with integrated human subject and intellectual property rights markup will both increase the efficiency of existing studies and support the growth of large, online education data sets that will allow researchers to engage in much-needed secondary analysis.

Educational Outcomes

The Digital Insight partnership - WCER, TalkBank, and SDSC - represents a broad range of skills, technologies, and user communities. We have the capacity to improve the productivity of researchers in many areas. The pressing needs of math and science educators will be well served by this research and by the output of the participating research teams.

Support for New Theories of Learning

This project will support ongoing research in the cognitive sciences that explores the importance of contingency and history in learning. Much of the current research in mathematics and science education, for example, seeks first to design effective learning environments and then to characterize resulting developmental trajectories of learning (Cobb, 2001; Lehrer & Chazan, 1998; Schauble & Glaser, 1996). This form of research relies on distillation of longitudinal video, still-image, and observational data to create rich models of learning-in-context, with specific attention to interactions among tasks, discourse, and systems of representation in classroom settings. The units of analysis within such frameworks are various forms of mediated activity that are typically represented by coordinations among tools, inscriptions, and language (Wertsch, 1998). Models of learning-in-context not only describe typical milestones of conceptual accomplishment, but also seek to uncover how students learn to participate in discipline-centered forms of argument and representation. These new efforts aim to create a theory of understanding to complement current theories of skill in cognitive science (Greeno, 1998).

New Forms of Professional Development

New theories of learning provide challenges and opportunities for the preparation of teachers. Current efforts to prepare teachers usually fail to help prospective practitioners integrate knowledge of student learning with conditions of practice. Current teacher preparation tends to emphasize learning that is divorced from context and to describe contexts (like cooperative groups) in ways that are divorced from learning. Consequently, teacher educators are exploring alternative forums for professional development. One of the most promising avenues is the construction and use of case-based forms of instruction (Derry, Levin, Osana, Jones, & Peterson, 2000; Lampert & Ball, 1998; Merseth, 1996; Williams, 1992). Ideally, cases highlight the development of student thinking and the teaching practices that scaffold such development. New, richer conceptions of the links between psychological development and learning can be operationalized in clearly defined video sequences that will allow instructional designers to move toward a process-reengineering model of reform. Recent research exploring the nature of effective cases for professional development underscores the importance of realistic video because much of the activity of learning is more readily exemplified than defined (Horvath & Lehrer, 2000; Koehler & Lehrer, 1998). With tools provided by Digital Insight, video that is originally collected for education research sponsored by WCER and its partner institutions can be repurposed to inform the design of teacher education.

Improvements in Event-Based Modeling for Education Reform

The Digital Insight project will provide education researchers with tools for extracting event-based data from video streams and correlating these data with, for example, field notes, lesson plans, and assessment data. Digital Insight will include features such as causal grouping and event pattern interpretation that will improve information accessibility for case-based inquiry and analysis. Perhaps most important, Digital Insight will give an interdisciplinary team a way to characterize and analyze the operations of a project within a common framework. In this way, a variety of perspectives will be able to be brought to bear in identifying and removing inefficiencies in the research process.

Collaborative Framework

This new project matches up with the three conditions that we have found to be important for any research and development collaborative project (Thorn, 1996; Willke, KrŁck, & Thorn, 1995):

1.      Leadership engaged in unconditional giving and focused on long-term gains;

2.      Identification of common, challenging, basic research problems across the sector; and

3.      Cooperative goals that do not compete with or impede the individual missions of any collaborator.

The project leaders at the three major sites meet the first criterion for successful collaborative R&D efforts. The project teams have been pursuing the core goals of Digital Insight independent of any outside funding, with costs borne by their research institutions and by the PIs themselves. Furthermore, our progress to date in building prototypes and testing features has both encouraged the team to continue its efforts and, more important, sparked the interest of an increasing number of researchers who have been struggling with the challenges of engaging in large-scale video research.

The ubiquity of the large collection problem meets the second criterion of successful collaboratives. For example, at least one third of the research projects in WCER use video in some aspect of their research or dissemination efforts, and several maintain collections ranging from 500 to 12,000 hours of video. The most recent American Educational Research Association conference included 44 panels and symposia on the use of video in research and teaching. Portfolio projects at every level of the education system are using video to document student and instructor work. TIMSS-R has increased the visibility of video as a source of data for gaining additional insights from large-scale quantitative work (see, e.g., Stigler, Gonzales, Kawanaka, Knoll, & Serrano, 1999).

The product and research goals of the Digital Insight project meet the last criterion. The project does not compete with any of the research and dissemination deliverables of the participating research projects. Teams with radically different methodological and epistemological approaches need the same collection management and video analysis tools. Digital Insight will provide the infrastructure to allow multidisciplinary video-based research and teaching to become a reality.

Project Structure and Scope of Work

Digital Insight will provide a research process environment that offers the organizational and analytical tool sets for exploring and organizing event-based data and, in particular, building case histories from video data. The application of leading-edge video management technologies will allow researchers to study many more cases much more quickly than is possible with analog video.

Development Teams and Partnerships

The development teams represent a three-way partnership between WCER, TalkBank, and SDSC. At WCER, the development team will consist of professional staff from the Technical Services Department who specialize in video and Web product services, as well as staff members with information science and human-factors engineering experience. At SDSC, the development team will consist of the chief architect of the Storage Resource Broker and Metadata Catalog and the team of programmers working on making links to users who do not usually consume supercomputing resources (Moore, 2000). TalkBank participation will be through the PI at Carnegie Mellon, who has pioneered data sharing in the CHILDES (Child Language Data Exchange System; http://childes.psy.cmu.edu/) system, and through his development team, which is currently implementing a new suite of programs based on the ATLAS (Architecture and Tools for Linguistic Analysis Systems) data exchange under development by the National Institute of Standards and Technology and the University of Pennsylvania.[9]

Advisory Panel

The development team will also rely on a panel of outside experts to review the development of analytical tools and assist in prioritizing development goals. The advisory panel will be made up of video research experts, human-factors engineers, evaluators, and teacher educators. Potential advisory panel members include Norman Webb (University of Wisconsin-Madison), Jonathan Grudin (Microsoft Research), and Henry Thompson (University of Edinburgh, Scotland). We also plan to include international partners from the Institute for Interdisciplinary Research in Culture, Multimedia, Technology and Cognition based at the University of Prince Edward Island, Canada, and the Knowledge Media Research Center based at the University of TŁbingen, Germany.

Organizational Model for Collaborative Research and Development

Digital Insight will use an open-source model for collaborative research and development. In this model, a small core group of developers (e.g., WCER, TalkBank, and SDSC) agrees to a shared set of application design criteria and engages in user needs analysis and evaluation to support that goal. The core development teams then share their design goals and core documentation with a trusted group of supporters and potential users. These groups are given privileged access to the development activity and are encouraged to submit requests and changes and to engage in parallel development that will be submitted to the core developers and integrated into the project. The PIs and associated research staff on the Digital Insight research teams will make up this second level of the development model. Their students and colleagues will constitute the third level - a wider circle of end users. The participating research teams will filter the needs and interests of the third level and provide feedback to the core development teams. The development teams will also post source code on the project Web site and support related open-source developments.[10]

WCER will provide direct support for all participating research teams engaged in the Digital Insight AMAD process model and will conduct project evaluation and user-centered design study tasks. Programming will be done on-site at WCER and at Carnegie Mellon as specifications emerge for linking individual tools together. The teams will also collaborate with the developers at SDSC to ensure that the analytical tools, management environment, and dissemination tools support the design criteria for data exchange and analytical markup.

One important element that differentiates the Digital Insight project from the existing work being done on TIMSS-R is the reliance on an open data storage standard and free tools. We have examined the proprietary software vPrism as a possible platform for this work (see http://www.lessonlab.com/). However, the follow-on to the vPrism system (developed by LessonLab) focuses on building online course materials rather than conducting data-based, exploratory research. Data coded within vPrism systems are not then available in forms that will allow further data sharing or access over the Internet. Moreover, vPrism is prohibitively costly for smaller research efforts. LessonLab and other proprietary systems are designed for large-scale delivery of course materials and are priced for organizations prepared to purchase many client licenses. This level of investment decreases the likelihood that owners of new research and teaching collections would collaborate by sharing their video files, since the acquisition and management costs could never be recovered. The prohibitive cost and/or proprietary formats used in several of the available commercial tools - vPrism, motion analysis tools for improving sports performance, Code-A-Text, Qualitative Media Analysis, etc. - present a serious barrier to many researchers. Digital Insightís free tools and open standard for data exchange will lower the barriers to participation and encourage exploration and data collaboration.

Project Tasks

Digital Insightís research and development efforts will begin with the support of three funded research projects and one graduate student project. We will focus on research projects that have existing video data sets, that have developed a methodological framework for video analysis, and that attack different areas of education research. It is important in the early development phase that we minimize the transaction costs of coordinating the research and development teams so that we can scale up rapidly. In each of Years 2 and 3, we will add one or two additional projects. These additional projects will be selected based on the recommendations of our advisory panel as we work to make both the tools and the management environment available for testing to the widest possible range of users. Hosting working group meetings and participating in scholarly and technical conferences will provide an avenue for professional development and dissemination efforts, as well as a vital channel for feedback for both the design process and the associated research teams as they engage in new analysis and create new artifacts for dissemination. We also intend to develop an interdisciplinary working conference for research using video as a primary source. One of our long-term goals is to become an archival site for researchers interested in conducting secondary analysis of existing research video collections.

The projects chosen for Year 1 represent a range of approaches to research and differences in scale. The annual selection process will allow for a level of flexibility that should address any inadequacies in the initial selection and provide opportunities to include new and novel uses of video data. Offsite research teams will be provided with small subcontracts to offset the costs of participating. Onsite teams will receive in-kind support from the Digital Insight project staff.

Year 1 Project Research Teams

The project research teams will both inform the development work and benefit directly from the advances in analytical tools, data management environments, and dissemination products. By introducing existing tools in Year 1, we will be able to have an immediate impact on the participating research teams and also move more quickly into designing follow-on technologies.

Matthew Koehler (Michigan State) - Beyond CGI: Professional Development Materials for Geometry and Fractions. Koehlerís work focuses on the development of case-based hypermedia tools for teachersí professional development (Koehler, in press; Koehler & Lehrer, 1998). Specifically, Koehlerís work uses classroom video to build cases of early-grade mathematics and science instruction that make the "big ideas" (Bardeen & Lederman, 1998; Koehler, in press; Koehler & Lehrer, 1998) visible to teachers. A prominent feature of these hypermedia tools is the linkage between events in the video (e.g., instances of the "big ideas" in practice) and other resources in the knowledge base (explanatory text, expert commentary, related video cases, examples of student work, etc.).

Currently, Koehler is working with Lehrer and Schaubleís Modeling Nature project (see below) to develop case-based hypermedia tools that exemplify the development of model-based reasoning related to the topic of growth and diversity, one of the central strands in the national science standards. Products of this collaboration will be used in the professional development component of the Modeling Nature project, which involves some 14 teachers in five schools.

The contribution of Digital Insight to this work will be in the collection, storage, analysis, and indexing of the video, text, samples of student work, and other pieces that make up the knowledge base that will provide the foundation for the development of the case-based hypermedia tools. Even if all the work were to be done in one location, the integrated tools proposed by Digital Insight would greatly facilitate the development of hypermedia tools that feature tight linkages between elements of the knowledge base. However, most of the work of the Koehler project will be done collaboratively; video, text, and samples of student work will be selected and analyzed across multiple research sites. The tools proposed by Digital Insight will make this collaboration much more effective. If the work proposed by Koehler for the Phoenix and Minneapolis school districts is funded, Digital Insight will have a key role in connecting researchers and teachers as they collaborate on the development of professional development materials.

Richard Lehrer and Leona Schauble (Vanderbilt) - Modeling Nature: Tackling the Challenges of Standards-Based Instruction. One of the foci of Lehrerís and Schaubleís NSF-funded work on early-grade math and science education is the construction of a knowledge base to systematically study the conceptual development of students as they move through years of instruction designed to emphasize consistency and coherence with regard to central themes or "big ideas"(Bardeen & Lederman, 1998). Since understanding of these organizing themes does not emerge over a few months or even a year, this work needs to be conducted in settings where teachers and administrators are committed to this approach in order to build histories of learning (Lehrer & Curtis, 2000). In other work funded by NSF, the James S. McDonnell Foundation, and the U.S. Department of Educationís Office of Educational Research and Improvement, Lehrer and Schauble have been collaborating with four public elementary schools to develop a research-based approach to reforming elementary school mathematics and science instruction.

Digital Insight will enhance this projectís ability to build a research base to support instruction that is guided by knowledge about how student understanding typically unfolds. It will support Lehrerís and Schaubleís goal of discovering and documenting the conceptual resources and barriers that students bring to the learning process and identifying the curricular and instructional strategies that are most effective in fostering conceptual development.

The Modeling Nature project requires analytical tools that permit variations in the granularity of analysis. Such tools are important when one examines histories of learning, because fine-grained analysis of individualsí strategies and representations is often governed by larger grained considerations like classroom norms and related teaching practices. The tools must allow researchers to make conjectures and tests of systematic relations between collective and individual planes of activity (Strom, Kemeny, Lehrer, & Forman, 2001). During the first year of the project, Lehrer and Schauble propose to characterize the history of fourth and fifth gradersí ideas about distributions as signatures of growth as these unfold over the course of approximately 60 hours of classroom activity. The analysis will extend a graph-theory approach to characterizing the collective structure of mathematical argument developed by Strom et al. (in press) and will also characterize individual cases of learning in relation to this collective structure. Digital Insightís management environment will also allow project researchers to identify those elements of their findings that can be represented using multimedia professional development materials and that therefore do not rely solely on face-to-face consulting and mentoring. The research team will use Digital Insight tools to create these materials and disseminate these findings using print, video, and Web-based technologies. Because Lehrer and Schauble are moving from WCER to Vanderbilt, they will also engage in building capacity for digital video analysis at Vanderbilt with a team of mathematics and science educators. This move is fortuitous because it provides an opportunity to explore the transport of the AMAD model to a new institution with colleagues who share a vision about mathematics and science education but whose work has not previously been oriented toward building systemic capacity in digital video. Consequently, we also propose to document steps in the development of this collaborative infrastructure and to conduct periodic interviews with its participants.

Sharon Derry (UW-Madison) - Secondary Education Teacher Preparation: Evaluating Cases and Case Design. There is a growing trend toward using video cases in teacher professional development because cases can provide a rich instructional context. However, little research has been conducted on how teachers learn from cases and how case design affects learning. The Secondary Teacher Education Project (STEP) is an NSF-funded program of research and development that is addressing these issues. The STEP program is developing and investigating an innovative online form of case-based teacher professional development offered through the STEP Web site (http://estepweb.org). To enhance the instructional effectiveness of its Web site, the STEP team needs to develop additional video cases. These new cases will represent a broad range of teaching situations, instructional problems, techniques, and subject disciplines. There are few existing video collections that both suit STEPís purposes and are legally available for display on the Internet. To build these new cases, the STEP team is videotaping teachersí instruction in the classroom, interviewing teachers, and obtaining supporting instructional materials. Case development is being conducted collaboratively with Rutgers University, Michigan State, and the Georgia Institute of Technology, with funding from the Joyce Foundation.

STEP is also investigating whether perceptual enhancements that employ color, sound, and imagery to emphasize learning-science themes in video cases can be used to highlight complex thematic interactions in video cases, helping learners acquire the ability to think flexibly and interpret classroom interactions from multiple perspectives. STEP is conducting this research collaboratively with Michigan State University.

Digital Insight will provide the critical infrastructure and process model necessary to allow multisite, multidisciplinary case building and analysis to become a reality. The components of the Digital Insight system that will serve the needs of STEP are:

1.      Web-based support for collaborative viewing, coding, segmenting, and combining of video;

2.      The capability to enhance video with symbolic overlays and codes; and

3.      The capability to store and support dynamic assembly and reassembly of coded, enhanced video segments to produce customized cases for objectives-based teacher professional development.

Chris Fassnacht (UW-Madison) - The Interactive Constitution of Childrenís Cognitive Abilities.[11] Fassnacht will analyze videotaped sessions in which K-6 children suspected of having cognitive deficiencies are examined using standard test instruments applied in an interactive setting. Subsequent tapings of sessions in which the assessments of the childrenís cognitive abilities are delivered to their parents will also be studied. Using conversation analytic (Sacks & Jefferson, 1992) and ethnomethodological (Garfinkel, 1967) approaches to these recorded data, Fassnacht will attempt to uncover some of the collaborative structures by which objective measures of cognitive capacity are distilled from an interactional foundation of testing, assessment, professional opinion, and everyday interaction. One possible outcome of the study is the elucidation of those structures of interaction whereby individuals are categorized according to lay and professional knowledge of what constitutes evidence of normal and abnormal cognitive ability. Digital Insight will support the digitizing, storage, and management of the video data collection (approximately 50 hours of tests and meetings). In addition, Digital Insight will provide access to advanced software tools for qualitative data analysis and grounded theory building.

Years 2 and 3 Project Research Teams

In each of Years 2 and 3, the Digital Insight project will extend its tools and support efforts to two additional research teams selected from a pool of potential candidates, including the TalkBank Classroom Discourse Working Group (see, e.g., MacWhinney & Snow, 1999) and other researchers identified by the advisory panel as doing compelling work supporting high-impact learning in early grades using video-based research. To date, the following scholars have indicated their interest in becoming a participating research team: Rogers Hall (Vanderbilt), Jim Kaput (UMass-Dartmouth), Richard Lesh (Purdue), Rand Spiro (Michigan State), and Deborah Vandell (UW-Madison).

Project Deliverables

Teacher Education and Professional Development

The primary deliverables for the project will be both the development teamís analysis of best practices for collaborative video research and the substantive work that the new technologies will allow participating research teams to produce. The ability to intelligently manage and analyze complex curricula will be greatly enhanced. The ability to collaborate over distance within a secure environment will both enable the development of new forms of teaming and address the concerns of research participants about the security of the data. We will share results of our own research on best-practice processes and tools with all participating research teams and the larger academic community through presentations, applied workshops, and scholarly writing.

Analytical Tools

The Digital Insight proposal is designed to take advantage of the existence of prototype analytical tools for video analysis and ongoing efforts to build data exchange and cataloging techniques. We will be able to incorporate analytical models, markup languages, and output features directly into our user needs analysis. Our user communities already recognize the functionality and value of many of these existing tools. We will take advantage of work done with commercial qualitative analysis tools such as NUD*IST, ATLAS.ti, NVivo, and Ethnograph to deliver an open-coding system that can support a wide range of theoretical and methodological approaches.

Storage Management: Cataloging and Collaborating Over Distance

The Storage Resource Broker and Metadata Catalog developed by the SDSCís Data-Intensive Computing Environments team are well adapted to an audience that is technically sophisticated and engaged in remote computing. We will also explore the feasibility of adapting the Metadata Encoding & Transmission Standard (METS) markup schema for use within Digital Insight. The importance of a standard cataloging system cannot be overestimated, particularly for large, long-term projects. The TalkBank proposal addresses the problem at the individual researcher level through the development of the ATLAS data exchange standard. Digital Insight will approach the problem from the institutional level and will include best practices for the management of complex collections.

Human Subjects: Support for Secondary Analysis and Embedded Permissions

One of the most important - and often overlooked - aspects of data acquisition is the creation and management of human subject use releases that adequately describe both what analysis will be done with the data and how access will be controlled to prevent unacceptable uses. The Digital Insight team will work with researchers and institutional review boards and will rely on the expert opinions of the advisory panel to identify the important dimensions of permission, use, and repurposing of data. Human subject deliverables will include both technical solutions to better management of use rights through ensuring that rights are embedded in the content management system and the development of consent forms that describe best practices for using video research data.

User-Centered Design

Digital Insightís integrated user-centered design research approach will provide ongoing feedback on the development process and describe changes in research methodologies employed by participating teams. The advisory panel will support this work at their annual meetings where they will review the research and evaluation progress, act as outside reviewers of the development teamís scholarly efforts, and make suggestions for midcourse corrections in the design and development efforts. The user-centered design and formative evaluation deliverables will be tightly integrated to provide data for both internal feedback and external accountability.

Summative Evaluation Studies (CIPP)

We will adapt the Digital Insight summative evaluation process from the CIPP (context, input, process, product) framework originally developed by Stufflebeam in the early 1970s to evaluate projects that were not conducive to traditional evaluation and experimental design methods (Stufflebeam, 1971; Stufflebeam, 2001). The CIPP framework has been used in a variety of ways over the last 3 decades to provide ongoing feedback to program implementation, to serve as a model for systematizing methodological approaches, and to identify process- and product-oriented indicators of quality (Hekimian, 1984; Madaus, Scriven, & Stufflebeam, 1983; Metzler & Tjeerdsma, 1998; Seyfried, 1998; Trevisan, 1997). The CIPP framework frees the evaluation design from assessing the degree to which vague objectives have been met, allowing the focus to shift to assessing process and implementation, stimulating planning and changes, and providing ongoing and useful information for decision making.

Digital Insight and ROLE

We recognize that the proposal budget exceeds the amount typically requested for ROLE funding. It is our belief, however, that ROLE is the most appropriate program for this work and that broad scope and scale of the work is vital if one is to test the efficacy and improve the output of collaborative research. The multi-institutional and cross-disciplinary research supported within and by Digital Insight could have a profound impact across much of the EHR portfolio.

Results from Prior and Current NSF Support

Christopher Thorn

Thorn is currently PI on two subgrants from NSF-supported projects. He is in the 3rd year of a subgrant from the National Partnership for Advanced Computational Infrastructure (ACI-9619020) to develop prototype tools for accessing the SDSC-based Storage Resource Broker. He is also in the 2nd year of a subgrant from the TalkBank project (BCS-9980009) to participate in the development of an open-source video analysis tool.

Arcot Rajasekar

ITR: Exploring the Environment in Time: Wireless Networks & Real-Time Management - Award Number 0121726. Start, October 1, 2001 - Expires, November 30, 2004 Total Amount $1,758,079.

KDI: A Knowledge Network for Biocomplexity: Building and Evaluating a Metadata-Based Framework for Integrating Heterogeneous Scientific Data: Award number 9980154; September 15, 1999-July 31, 2002; expected total amount $2,979,000.

Brian MacWhinney

Tools for the Linguistic Analysis of the CHILDES Database: Award number 9808974; September 15, 1998-August 31, 2001; expected total amount $300,000.

KDI: TalkBank: A Multimodal Database of Communicative Interaction: Award number 9980009; September 15, 1999-August 31, 2002; expected total amount $1,442,000.



[1] AMAD (described more fully on page 5) stands for acquisition, management, analysis, and dissemination. It is this integrated, end-to-end process model that ties the Digital Insight technologies and research goals together in a unified design and development activity.

[2] See the link to JLS-Chat at http://talkbank.org/data/class.

[3] For more on CLAN, visit http://childes.psy.cmu.edu/.

[4] For more on Transana, visit http://www.transana.org. We are currently developing open-source versions of the software for both Windows and OS X.

[5] This is some danger of control groupsí being contaminated by association with colleagues using Digital Insight tools at the same site or simply having access to the tools over the Internet. In any case, the initial selection will provide a wait list for staggered admission as an alternative. Although this approach will diminish the control, it will also provide greater insights into the dynamics of technology adoption.

[6] See http://www.talkbank.org/share/levels.html for a multitiered access model. See also MacWhinney & Snow (1999) for an analysis of the levels of release necessary for a shared database of video on instructional interactions.

[7] Federated storage systems are a collection of disparate storage devices that are owned and operated by different agencies. These resources are tied together with a distributed management architecture that provides integrated data security, network management, and redundant storage. For more on federated systems, see Rajasekar, Marciano, and Moore (1999).

[8] For more on the economics and dynamics of open source, see Lerner and Triole (2000) and Perens (1999).

[9] ATLAS will provide a basic application program interface (API) that will support the exchange of data between different analytical tools using a device known as annotation graphs. See http://agtk.sourceforge.net/.

[10] The Xcode (a spin-off tool from the TalkBank project) and Annotation Graph Toolkit developed by TalkBank are currently available at the open-source code repository - SourceForge (http://agtk.sourceforge.net/).

[11] Fassnacht has also been active in research on the use of video to study classroom interaction (Zuengler, Ford, & Fassnacht, 1999).