The
ubiquity and instructional power of video in teacher professional
development, studies of classroom practice, and preservice teacher
education ensure a broad target audience for the Digital Insight
tools and processes.
Range
of Use
Models
Use
models for Digital Insight range along a number of dimensions.
Three of the most important dimensions are the granularity of the
analysis, the temporal resolution of the driving research questions,
and the size of the data set to be analyzed. Potential uses of the
Digital Insight tools and processes range from fine-grained
studies of human interaction (e.g., taking turns in conversation)
to large, ethnographic studies of change over years of instruction.
One particularly relevant example is a special edition of The
Journal of the Learning Sciences (Sfard & McClain eds.,
2002) in which six different scholars apply their particular theoretical
approaches to the same video of seventh graders’ being introduced
to statistical concepts. Brian MacWhinney extends this work by providing
an example of what he calls collaborative analysis,
in which he aggregates the analytical markup of the six authors
to give a combined, comparative view of the various methodical approaches.
Other
researchers use video in traditional A-B comparisons of various
treatments. This approach usually takes the form of studies of control
and treatment classroom video that covers the same instructional
area but contrasts different pedagogical techniques or access to
different resources. The Third International Mathematics and Science
Study-Repeat (TIMSS-R) study falls into the larger end of such mid-range
studies. Rather than a controlled experiment, however, TIMSS-R provides
a quasi-experiment by examining variation within and between classrooms
and within and between participating countries. Other large-scale
studies, such as Rich Lehrer’s work on building long-term histories
of individual students’ cognitive development across a number of
years, take quite a different approach. Lehrer’s research combines
both fine-grained analysis of case study students’ speech and numerous
historical accounts of the students’ work. Lehrer then compares
and contrasts these individual longitudinal studies to explore the
impact of both natural variance and variance in treatment.
Digital
Insight’s tools
will be relevant across this broad range of approaches - from individual
researchers and teams engaged in collaborative analysis to long-term
studies that span many sites and cross institutional boundaries.
Use
and Enhancement of Existing Technologies
The
Digital Insight project will build on the work of SDSC to
extend the benefits of its innovative analytical tools to education
researchers. The secure data storage and management environment,
called the Storage Resource Broker (Rajasekar, Wan, & Moore,
2002; Rajasekar & Wan, 2002; Rajasekar & Moore, 2001) and
developed at SDSC will provide Internet-based access to large, complex
collections of research video and other related evidence. The infrastructure
being developed by SDSC reflects a rich understanding of cross-disciplinary
research and represents a great capacity for helping organizations
understand ‘how they manage what they know.’ (Davenport & Prusak,
1998). Currently, the most ambitious project to develop a standard
for managing, exploring, and creating links within heterogeneous
collections of multimedia data is being conducted by the Library
of Congress and its partners within the Digital Library Federation
(http://www.diglib.org/). The
Library of Congress is already a collaborator with SDSC on a prototype
project to operationalize distributed collection management using
the SDSC-based storage grid. We will take advantage of the work
that has already been done on federating and searching complex collections.
The
TalkBank development team will work with WCER programmers on the
creation of open-source tools for video transcription, linkage,
and analysis that follow from the work they have done in the area
of analysis of child language. CLAN, their transcript analysis
tool, has been an important tool for linguistic analysis across
the globe.
In particular, the markup language CHAT that grew out of
the use of CLAN has become a dominant form of linguistic
markup and is supported by many of the other analytical tools available
for linguistic research. The TalkBank team’s experience in developing
and supporting analytical software for research in the social sciences
will provide a vital backdrop as we move towards a more complex
use environment and model.
Transana's
Main screen:
WCER
brings to the table a tool for video analysis - Transana.
Transana is available for both single users and research teams
and is under active development. Transana is designed to
facilitate the transcription and analysis of video data. It provides
a way to view video, create a transcript, and link places in the
transcript to frames in the video. It provides tools for identifying
and organizing analytically interesting portions of videos, as well
as for attaching keywords to those video clips. It also features
database and file manipulation tools that facilitate the organization
and storage of large collections of digitized video.
Transana
consists of a Sound Waveform Window, a Video Window, a Transcript
Window, and a Database Tree. The Sound Waveform presents a visual
representation of the audio track of the video being analyzed, allowing
great precision in selecting portions of the video for analysis.
The Transcript Window is a word-processing window that allows users
to easily create and edit transcripts for videos. A frame in the
video can be linked to a position in the transcript by inserting
a time code. Such links enable Transana to synchronize video
playback with portions of the transcript. The Database Tree gives
a visual representation of the organization of data within Transana,
making it easy to manage even large collections of analytically
interesting video clips. Analytic keywords can be assigned to segments
of video data, and Transana’s search features allow for increasingly
sophisticated data mining. One can then group video clips to assist
in the analysis of data.
Existing
Digital Insight tools, notably CLAN and Transana,
are and will remain available for free download to anyone interested
in the software. All improvements to this software and any additional
software developed will be made available as well - both in executable
form and as source code for those users who would like to make their
own extensions to the work. The only resource that will not be available
to research teams that are not active participants in the Digital
Insight project is access to the project’s large-scale storage
systems. However, one of the project’s goals is to provide a detailed
model for building local storage resources and integrating them
into a local Digital Insight-like infrastructure. We also
intend to extend the SDSC-based storage management system (Storage
Resource Broker [SRB]; http://www.npaci.edu/DICE/SRB)
to other research institutions. These relationships will be negotiated
on a case-by-case basis since they require a level of trust and
interoperation that exceeds that necessary for using the stand-alone
tools. In the meantime, the multiuser version of Transana
is already capable of supporting multisite collaboration. What it
lacks (and what Digital Insight will add) is the ability
to support more sophisticated collection management and search technologies
that are available within the SRB.
Methodology
Digital
Insight will research the applicability of new
technologies to a wide spectrum of education research projects.
Our selection of research efforts for study (see pp. 11-14) is intended
to sample from this spectrum. Accordingly, we will support the research
teams’ work through the AMAD process model as they transform their
multimedia data into scholarly and instructional deliverables. Our
primary technical focus will be to identify the operational characteristics
of the AMAD process and implement best-practice strategies in collaboration
with the research teams.
The
research design for this study is that of the quasi-experiment with
comparative cases. We will establish baseline characteristics of
both control groups (teams using analog case-building methods) and
treatment groups (participating research teams using digital case-building
methods) at the beginning of the project. The treatment will vary
according to which technologies are used in the acquisition, management,
analysis, and dissemination of research objects.
The comparative case study approach we will use is based on the
designs suggested by Barlow, Hayes, and Nelson-Gray (1998) and Kazdin
(1998) in their excellent outlines for clinical studies of comparative
cases involving scientist-practitioners. We will also take into
account the work of Gummesson (2000) as we examine the interaction
between academic researchers and consultants in qualitative case
studies. Collaboration between technical and substantive experts
will be an important aspect of the interaction between the programming
and research support consultants on the core development team and
the participating research teams.
We
will create case studies for each of the participating research
teams described below (pp. 11-14). The project staff includes researchers
with experience in ethnography, quantitative analysis, formative
and summative evaluation, workflow analysis, interface design, and
information system design. Data collection efforts will include
interviews with PIs, direct observation of research team meetings,
and documentation and analysis of the evolution of research teams’
analytical models as they incorporate Digital Insight tools
into their work. We will also document the interaction between the
research teams and the development team to study how best to coordinate
the work between researchers and those building research tools.
The
PI, Christopher Thorn, will lead and coordinate the ethnographic
study of the impact of Digital Insight on the scholarly processes
and outputs of the participating research teams. Sarah Mason, the
project manager, will lead the evaluation efforts and coordinate
the work of the library and information systems staff who will provide
assistance with interface and information system design. David Woods
will lead the software development efforts. His unusual background
in counseling psychology and analytical software design will be
an asset as he provides both software design and research support
to participating research teams. Woods will also support the ethnographic
study by conducting research staff interviews and using detailed
participant protocols that require the research teams to reflect
on their own work and the impact of Digital Insight tools
and process on their work. The Digital Insight project team
will provide ongoing feedback to the participating research teams
about our analysis of their work and the impact of our collaboration.
We will also provide a framework for cross-case comparisons that
will contrast the structural and behavioral similarities and differences
between the teams. These comparisons will provide the context for
analyzing the scholarly outcomes across the cases. Our goal is to
be able to describe several best-path models for the efficient use
of video data in research and teaching.
A
four-phase system of consecutive operations can describe the overall
process of most education research activities. These four operations
consist of the acquisition, management, analysis, and dissemination
of findings based on data (AMAD). This general model can be adapted
to any project.
The
importance of the AMAD model lies partly in two of its benefits:
it emphasizes phases of the research process that might otherwise
receive insufficient attention, and it treats the various operations
of education research as part of an integrated system of knowledge
production, thus allowing for synergistic improvements in the research
process at the system level. For example, the AMAD model will help
to clarify how acquiring data in a particular format affects not
only its management and analysis, but also the practical range of
possible products for dissemination. The usefulness of the model
is due both to its inherent applicability to a wide range of research
approaches and to its ability to be customized to fit each approach.
The model can accommodate a typical research project that simply
acquires, manages, analyzes, and disseminates data, but it can also
address the needs of projects that repeat the AMAD sequence or that
perform the AMAD operations in nonsequential order (e.g., analyze
the data as it is being acquired).
To
maximize end-product utility and usability, Digital Insight
will coordinate the development of technological tools (video analysis,
data management, and presentation software) with the development
of user practices within an iterative and user-centered design framework.
User-centered design principles help streamline product evolution
by providing developers with design recommendations throughout the
development cycle, especially during early development efforts when
significant changes can be made to design goals with minimal costs
(Baecker, Grudin, Buxton, & Greenberg, 1995; Nielsen, 2000;
Rouse, 1991; Shneiderman, 1998). Adopting user-centered design practices
will also help facilitate and maximize the benefits of interdisciplinary
work between development teams and research teams. The initial identification
of user groups and assessment of user needs will occur during the
first year of the project and will provide baseline information
for further design practices (Nielsen, 1994).
We
will construct process flow models of video-based education
research with the help of task analysis and other direct observation
techniques. These process flow models will (a) document the process
by which identified users actually go about conducting such research;
(b) allow us to target specific tasks as inefficient or otherwise
problematic; and (c) allow developers and researchers to calculate
metrics that can serve as tools for evaluating various methods.
We will also explore the use of appreciative intervention
to engage users in speculation about possible future uses of existing
tools and possible additional uses based on proposed feature set
changes (Karasti, 2001).
1.
How can we improve the use of video in education
research, and how will digitizing video and using the AMAD approach
enhance the research process?
We
will observe the work processes of researchers using traditional
analog methods and those using Digital Insight tools to build
video case histories. To help us establish a baseline for the differences
between these two sets of researchers, we will interview team members
about their research goals and their expectations about the workload
associated with the research process. In the case of our participating
teams, we will also debrief them about recommended refinements to
the tools and workflow/process models. The implementation of each
round of new tool developments will be followed by an additional
wave of field observations and interviews.
2.
How will the use of analytical tools for
combining video and transcript analysis - in addition to the inclusion
of other types of evidence (images of student work, audio, etc.)
- improve the research findings and their application for participating
research projects?
Researchers
will be asked to act as participant observers and complete regular
field logs recording problems encountered, suggestions, new insights,
the impact of the tools on the quality or quantity of their research
output, and similar observations. Members of the evaluation team
will also collect research output - for example, presentations,
papers, and articles - and compare them to previous work of the
same authors (when available) and to the work being done by control
group teams and their peers, as identified by each team. We will
also interview research team members to explore the impact of Digital
Insight tools on their scholarly work. We expect to see changes
in dissemination methods, an increasing sophistication of argumentation,
and the inclusion of new forms of evidence in scholarly debate.
3.
What impact will an integrated, secure,
Internet-based data management environment have on education research
conducted by geographically distributed, interdisciplinary teams?
The
increasing importance of supporting distributed teams is borne out
by the constitution and collaborative methods of the core team itself.
Secure, online collaborative tools will be essential for supporting
rapid, efficient software development and effective research workflow
processes across geographically dispersed teams. The same field
techniques described above will be used to document the coordination
of activities both within the control groups and within (and across)
the participating teams. We will explore the participating researchers’
impressions of (a) their own effectiveness working at a distance
and (b) the impact these technologies have on establishing and building
team cohesiveness and trust.
4.
What models for linking human subject confidentiality
requirements can be embedded within the Digital
Insight system?
Federal
guidelines for human subject protection and the emerging technical
requirements for securing clinical research environments are still
in flux. However, Digital Insight will provide an opportunity
to field-test the TalkBank model of informed consent for the use
of video research data. Teams will be encouraged to use and refine
the TalkBank model as they interact with their respective institutional
review boards and with their research subjects. Refinements to the
release model and to the practices for documenting consent and use
rights will be one of the important scholarly outcomes for the participating
universities.
5.
Is a collaborative process model derived
from basic research collaboratives in manufacturing transferable
to education research?
The
scope of Digital Insight is broad enough to provide some
systematic insights into the dynamics of successful collaborative
research teams. Recent trends toward large awards to multi-institution
research partnerships by the U.S. Department of Education and the
National Science Foundation suggest that both of these agencies
view collaborative research efforts as an important source of innovation.
We will ask our advisory panel to consider including research teams
from several of these large groups - such as the new NSF-sponsored
Centers for Learning and Teaching - when it recommends new research
teams for inclusion in Years 2 and 3. Data collected to answer Question
3 above will allow to us explore the conditions necessary for successful
research collaboration.
6.
What types of users and uses are best supported
by these new tools and procedures, and what types of users and uses
seem hard to influence?
In
interviews and observations over the life of the study, we will
be looking for indicators of system failure and research needs that
remain unmet. Research teams (including the core team) will be asked
to reflect on the barriers to implementation and the unanticipated
costs associated with participation in the project. The formative
evaluation of the introduction of new tools and processes will be
fed back into the design process and will provide information for
a summative impact analysis at the end of the project.
Project
Foci and Impact
Online
resources already developed by the Digital Insight team and
related information about our research partners can be found at
http://www.wcer.wisc.edu/digitalinsight.
Our programming efforts will center on the development of user-friendly
front-end tools that can take advantage of the federated storage
management environment developed by SDSC.
These front-end tools will include open-source tools
such as Transana for the analysis of transcripts synchronized
to video and audio files. The Digital Insight project will
provide education researchers with tools for extracting event-based
data from video streams and correlating these data with, for example,
field notes, images of student work, lesson plans, and assessment
data.
Within
the context of an advanced human-computer interface based on tools
currently under development, Digital Insight will include
features such as code-based causal grouping and event pattern interpretation.
Analysts will be able to apply unique coding schemes to portions
of the transcripts of a video collection, as well as to portions
of the video itself. This ability is vital for supporting an iterative
research process that allows for the sort of ongoing or termless
approach to inquiry suggested by Koschmann
(in press) and Koschmann, Kelson, Feltovich, and Barrows (1996).
The resulting analytical environment will provide researchers
with tools for theory building and hypothesis testing. This work
will take advantage of the research done by designers and critics
of groupware technologies. Of particular importance will
be the caveats presented by Grudin (1990, 1994) and cited in Choo,
Detlor, and Turnbull (2000, pp. 92-93) on the pitfalls of designing
without understanding and reflecting the incentive structure of
key users.
Digital
Insight will support multiple ROLE program themes.
First, by providing broad access to large-scale, distributed storage
and free, feature-rich, open-source analytical tools, Digital
Insight will allow for a rapid scale-up of projects engaged
in analysis and manipulation of digital video data focused on student
learning. By linking the analytical tools to the existing storage
grid development efforts under way at SDSC, Digital Insight will
enable participants to take advantage of the newest developments
in the NSF-sponsored TeraGrid program. This involvement will also
provide SDSC developers with a broad, education-oriented community
to take part in its participatory design efforts.
Second,
Digital Insight’s impact on improving human capacity through
education research and instruction will be profound and transformative.
The construction of histories of learning is a vital step
toward developing and testing new, high-strength instructional approaches.
The work of the participating research teams is aimed at supporting
early acquisition of foundational skills and improving students’
transition to increasingly complex science and mathematics learning.
In particular, the teams are working on increasing the analytical
capacity of teams doing work on cognitive testing and development,
early math education, preservice teacher training in math and science,
and training of teacher educators.
Finally,
Digital Insight’s focus on the AMAD process model has the
potential to provide new insights into the dynamics of multisite,
interdisciplinary research practices. The introduction of a rich
collection management architecture with integrated human subject
and intellectual property rights markup will both increase the efficiency
of existing studies and support the growth of large, online education
data sets that will allow researchers to engage in much-needed secondary
analysis.
Educational
Outcomes
The
Digital Insight partnership - WCER, TalkBank, and SDSC -
represents a broad range of skills, technologies, and user communities.
We have the capacity to improve the productivity of researchers
in many areas. The pressing needs of math and science educators
will be well served by this research and by the output of the participating
research teams.
This
project will support ongoing research in the cognitive sciences
that explores the importance of contingency and history in learning.
Much of the current research in mathematics and science education,
for example, seeks first to design effective learning environments
and then to characterize resulting developmental trajectories
of learning (Cobb, 2001; Lehrer & Chazan, 1998; Schauble &
Glaser, 1996). This form of research relies on distillation of longitudinal
video, still-image, and observational data to create rich models
of learning-in-context, with specific attention to interactions
among tasks, discourse, and systems of representation in classroom
settings. The units of analysis within such frameworks are various
forms of mediated activity that are typically represented by coordinations
among tools, inscriptions, and language (Wertsch, 1998). Models
of learning-in-context not only describe typical milestones of conceptual
accomplishment, but also seek to uncover how students learn to participate
in discipline-centered forms of argument and representation. These
new efforts aim to create a theory of understanding to complement
current theories of skill in cognitive science (Greeno, 1998).
New
theories of learning provide challenges and opportunities for the
preparation of teachers. Current efforts to prepare teachers usually
fail to help prospective practitioners integrate knowledge of student
learning with conditions of practice. Current teacher preparation
tends to emphasize learning that is divorced from context and to
describe contexts (like cooperative groups) in ways that are divorced
from learning. Consequently, teacher educators are exploring alternative
forums for professional development. One of the most promising avenues
is the construction and use of case-based forms of instruction (Derry,
Levin, Osana, Jones, & Peterson, 2000; Lampert & Ball, 1998;
Merseth, 1996; Williams, 1992). Ideally, cases highlight the development
of student thinking and the teaching practices that scaffold such
development. New, richer conceptions of the links between psychological
development and learning can be operationalized in clearly defined
video sequences that will allow instructional designers to move
toward a process-reengineering model of reform. Recent research
exploring the nature of effective cases for professional development
underscores the importance of realistic video because much of the
activity of learning is more readily exemplified than defined (Horvath
& Lehrer, 2000; Koehler & Lehrer, 1998). With tools provided
by Digital Insight, video that is originally collected for
education research sponsored by WCER and its partner institutions
can be repurposed to inform the design of teacher education.
The
Digital Insight project will provide education researchers
with tools for extracting event-based data from video streams and
correlating these data with, for example, field notes, lesson plans,
and assessment data. Digital Insight will include features
such as causal grouping and event pattern interpretation that will
improve information accessibility for case-based inquiry and analysis.
Perhaps most important, Digital Insight will give an interdisciplinary
team a way to characterize and analyze the operations of a project
within a common framework. In this way, a variety of perspectives
will be able to be brought to bear in identifying and removing inefficiencies
in the research process.
Collaborative
Framework
This
new project matches up with the three conditions that we have found
to be important for any research and development collaborative project
(Thorn, 1996; Willke, Krück, & Thorn, 1995):
1.
Leadership engaged in unconditional giving and focused
on long-term gains;
2.
Identification of common, challenging, basic research problems
across the sector; and
3.
Cooperative goals that do not compete with or impede the
individual missions of any collaborator.
The
project leaders at the three major sites meet the first criterion
for successful collaborative R&D efforts. The project teams
have been pursuing the core goals of Digital Insight independent
of any outside funding, with costs borne by their research institutions
and by the PIs themselves. Furthermore, our progress to date in
building prototypes and testing features has both encouraged the
team to continue its efforts and, more important, sparked the interest
of an increasing number of researchers who have been struggling
with the challenges of engaging in large-scale video research.
The
ubiquity of the large collection problem meets the second
criterion of successful collaboratives. For example, at least one
third of the research projects in WCER use video in some aspect
of their research or dissemination efforts, and several maintain
collections ranging from 500 to 12,000 hours of video. The most
recent American Educational Research Association conference included
44 panels and symposia on the use of video in research and teaching.
Portfolio projects at every level of the education system are using
video to document student and instructor work. TIMSS-R has increased
the visibility of video as a source of data for gaining additional
insights from large-scale quantitative work (see, e.g., Stigler,
Gonzales, Kawanaka, Knoll, & Serrano, 1999).
The
product and research goals of the Digital Insight project
meet the last criterion. The project does not compete with any of
the research and dissemination deliverables of the participating
research projects. Teams with radically different methodological
and epistemological approaches need the same collection management
and video analysis tools. Digital Insight will provide the
infrastructure to allow multidisciplinary video-based research and
teaching to become a reality.
Project
Structure and Scope of Work
Digital
Insight will provide a research process environment
that offers the organizational and analytical tool sets for exploring
and organizing event-based data and, in particular, building case
histories from video data. The application of leading-edge video
management technologies will allow researchers to study many more
cases much more quickly than is possible with analog video.
The
development teams represent a three-way partnership between WCER,
TalkBank, and SDSC. At WCER, the development team will consist of
professional staff from the Technical Services Department who specialize
in video and Web product services, as well as staff members with
information science and human-factors engineering experience. At
SDSC, the development team will consist of the chief architect of
the Storage Resource Broker and Metadata Catalog and the team of
programmers working on making links to users who do not usually
consume supercomputing resources (Moore, 2000). TalkBank participation
will be through the PI at Carnegie Mellon, who has pioneered data
sharing in the CHILDES (Child Language Data Exchange System; http://childes.psy.cmu.edu/)
system, and through his development team, which is currently implementing
a new suite of programs based on the ATLAS (Architecture and Tools
for Linguistic Analysis Systems) data exchange under development
by the National Institute of Standards and Technology and the University
of Pennsylvania.
The
development team will also rely on a panel of outside experts to
review the development of analytical tools and assist in prioritizing
development goals. The advisory panel will be made up of video research
experts, human-factors engineers, evaluators, and teacher educators.
Potential advisory panel members include Norman Webb (University
of Wisconsin-Madison), Jonathan Grudin (Microsoft Research), and
Henry Thompson (University of Edinburgh, Scotland). We also plan
to include international partners from the Institute for Interdisciplinary
Research in Culture, Multimedia, Technology and Cognition based
at the University of Prince Edward Island, Canada, and the Knowledge
Media Research Center based at the University of Tübingen, Germany.
Digital
Insight will use an open-source model for collaborative
research and development. In this model, a small core group of developers
(e.g., WCER, TalkBank, and SDSC) agrees to a shared set of application
design criteria and engages in user needs analysis and evaluation
to support that goal. The core development teams then share
their design goals and core documentation with a trusted group of
supporters and potential users. These groups are given privileged
access to the development activity and are encouraged to submit
requests and changes and to engage in parallel development that
will be submitted to the core developers and integrated into the
project. The PIs and associated research staff on the Digital
Insight research teams will make up this second level
of the development model. Their students and colleagues will constitute
the third level - a wider circle of end users. The participating
research teams will filter the needs and interests of the third
level and provide feedback to the core development teams. The development
teams will also post source code on the project Web site and support
related open-source developments.
WCER
will provide direct support for all participating research teams
engaged in the Digital Insight AMAD process model and will
conduct project evaluation and user-centered design study tasks.
Programming will be done on-site at WCER and at Carnegie Mellon
as specifications emerge for linking individual tools together.
The teams will also collaborate with the developers at SDSC to ensure
that the analytical tools, management environment, and dissemination
tools support the design criteria for data exchange and analytical
markup.
One
important element that differentiates the Digital Insight
project from the existing work being done on TIMSS-R is the reliance
on an open data storage standard and free tools. We have examined
the proprietary software vPrism as a possible platform for this
work (see http://www.lessonlab.com/).
However, the follow-on to the vPrism system (developed by LessonLab)
focuses on building online course materials rather than conducting
data-based, exploratory research. Data coded within vPrism systems
are not then available in forms that will allow further data sharing
or access over the Internet. Moreover, vPrism is prohibitively costly
for smaller research efforts. LessonLab and other proprietary systems
are designed for large-scale delivery of course materials and are
priced for organizations prepared to purchase many client licenses.
This level of investment decreases the likelihood that owners of
new research and teaching collections would collaborate by sharing
their video files, since the acquisition and management costs could
never be recovered. The prohibitive cost and/or proprietary formats
used in several of the available commercial tools - vPrism, motion
analysis tools for improving sports performance, Code-A-Text, Qualitative
Media Analysis, etc. - present a serious barrier to many researchers.
Digital Insight’s free tools and open standard for data exchange
will lower the barriers to participation and encourage exploration
and data collaboration.
Digital
Insight’s research and development efforts will
begin with the support of three funded research projects and one
graduate student project. We will focus on research projects that
have existing video data sets, that have developed a methodological
framework for video analysis, and that attack different areas of
education research. It is important in the early development phase
that we minimize the transaction costs of coordinating the research
and development teams so that we can scale up rapidly. In each of
Years 2 and 3, we will add one or two additional projects. These
additional projects will be selected based on the recommendations
of our advisory panel as we work to make both the tools and
the management environment available for testing to the widest possible
range of users. Hosting working group meetings and participating
in scholarly and technical conferences will provide an avenue for
professional development and dissemination efforts, as well as a
vital channel for feedback for both the design process and the associated
research teams as they engage in new analysis and create new artifacts
for dissemination. We also intend to develop an interdisciplinary
working conference for research using video as a primary source.
One of our long-term goals is to become an archival site for researchers
interested in conducting secondary analysis of existing research
video collections.
The
projects chosen for Year 1 represent a range of approaches to research
and differences in scale. The annual selection process will allow
for a level of flexibility that should address any inadequacies
in the initial selection and provide opportunities to include new
and novel uses of video data. Offsite research teams will be provided
with small subcontracts to offset the costs of participating. Onsite
teams will receive in-kind support from the Digital Insight project
staff.
The
project research teams will both inform the development work and
benefit directly from the advances in analytical tools, data management
environments, and dissemination products. By introducing existing
tools in Year 1, we will be able to have an immediate impact on
the participating research teams and also move more quickly into
designing follow-on technologies.
Matthew
Koehler (Michigan State) - Beyond
CGI: Professional Development Materials for Geometry and Fractions.
Koehler’s work focuses on the development of case-based hypermedia
tools for teachers’ professional development (Koehler, in press;
Koehler & Lehrer, 1998). Specifically, Koehler’s work uses classroom
video to build cases of early-grade mathematics and science instruction
that make the "big ideas" (Bardeen & Lederman, 1998; Koehler,
in press; Koehler & Lehrer, 1998) visible to teachers. A prominent
feature of these hypermedia tools is the linkage between events
in the video (e.g., instances of the "big ideas" in practice) and
other resources in the knowledge base (explanatory text, expert
commentary, related video cases, examples of student work, etc.).
Currently,
Koehler is working with Lehrer and Schauble’s Modeling Nature project
(see below) to develop case-based hypermedia tools that exemplify
the development of model-based reasoning related to the topic of
growth and diversity, one of the central strands in the national
science standards. Products of this collaboration will be used in
the professional development component of the Modeling Nature project,
which involves some 14 teachers in five schools.
The
contribution of Digital Insight to this work will be in the
collection, storage, analysis, and indexing of the video, text,
samples of student work, and other pieces that make up the knowledge
base that will provide the foundation for the development of the
case-based hypermedia tools. Even if all the work were to be done
in one location, the integrated tools proposed by Digital Insight
would greatly facilitate the development of hypermedia tools that
feature tight linkages between elements of the knowledge base. However,
most of the work of the Koehler project will be done collaboratively;
video, text, and samples of student work will be selected and analyzed
across multiple research sites. The tools proposed by Digital
Insight will make this collaboration much more effective. If
the work proposed by Koehler for the Phoenix and Minneapolis school
districts is funded, Digital Insight will have a key role
in connecting researchers and teachers as they collaborate on the
development of professional development materials.
Richard
Lehrer and Leona Schauble (Vanderbilt) - Modeling Nature: Tackling
the Challenges of Standards-Based Instruction.
One of the foci of Lehrer’s and Schauble’s NSF-funded work on early-grade
math and science education is the construction of a knowledge base
to systematically study the conceptual development of students as
they move through years of instruction designed to emphasize consistency
and coherence with regard to central themes or "big ideas"(Bardeen
& Lederman, 1998). Since understanding of these organizing themes
does not emerge over a few months or even a year, this work needs
to be conducted in settings where teachers and administrators are
committed to this approach in order to build histories of learning
(Lehrer & Curtis, 2000). In other work funded by NSF, the James
S. McDonnell Foundation, and the U.S. Department of Education’s
Office of Educational Research and Improvement, Lehrer and Schauble
have been collaborating with four public elementary schools to develop
a research-based approach to reforming elementary school mathematics
and science instruction.
Digital
Insight will enhance this project’s ability to
build a research base to support instruction that is guided by knowledge
about how student understanding typically unfolds. It will support
Lehrer’s and Schauble’s goal of discovering and documenting the
conceptual resources and barriers that students bring to the learning
process and identifying the curricular and instructional strategies
that are most effective in fostering conceptual development.
The
Modeling Nature project requires analytical tools that permit variations
in the granularity of analysis. Such tools are important when one
examines histories of learning, because fine-grained analysis of
individuals’ strategies and representations is often governed by
larger grained considerations like classroom norms and related teaching
practices. The tools must allow researchers to make conjectures
and tests of systematic relations between collective and individual
planes of activity (Strom, Kemeny, Lehrer, & Forman, 2001).
During the first year of the project, Lehrer and Schauble propose
to characterize the history of fourth and fifth graders’ ideas about
distributions as signatures of growth as these unfold over the course
of approximately 60 hours of classroom activity. The analysis will
extend a graph-theory approach to characterizing the collective
structure of mathematical argument developed by Strom et al. (in
press) and will also characterize individual cases of learning in
relation to this collective structure. Digital Insight’s
management environment will also allow project researchers to identify
those elements of their findings that can be represented using multimedia
professional development materials and that therefore do not rely
solely on face-to-face consulting and mentoring. The research team
will use Digital Insight tools to create these materials
and disseminate these findings using print, video, and Web-based
technologies. Because Lehrer and Schauble are moving from WCER to
Vanderbilt, they will also engage in building capacity for digital
video analysis at Vanderbilt with a team of mathematics and science
educators. This move is fortuitous because it provides an opportunity
to explore the transport of the AMAD model to a new institution
with colleagues who share a vision about mathematics and science
education but whose work has not previously been oriented toward
building systemic capacity in digital video. Consequently, we also
propose to document steps in the development of this collaborative
infrastructure and to conduct periodic interviews with its participants.
Sharon
Derry (UW-Madison) - Secondary Education Teacher Preparation: Evaluating
Cases and Case Design.
There is a growing trend toward using video cases in teacher professional
development because cases can provide a rich instructional context.
However, little research has been conducted on how teachers learn
from cases and how case design affects learning. The Secondary Teacher
Education Project (STEP) is an NSF-funded program of research and
development that is addressing these issues. The STEP program is
developing and investigating an innovative online form of case-based
teacher professional development offered through the STEP Web site
(http://estepweb.org). To enhance
the instructional effectiveness of its Web site, the STEP team needs
to develop additional video cases. These new cases will represent
a broad range of teaching situations, instructional problems, techniques,
and subject disciplines. There are few existing video collections
that both suit STEP’s purposes and are legally available for display
on the Internet. To build these new cases, the STEP team is videotaping
teachers’ instruction in the classroom, interviewing teachers, and
obtaining supporting instructional materials. Case development is
being conducted collaboratively with Rutgers University, Michigan
State, and the Georgia Institute of Technology, with funding from
the Joyce Foundation.
STEP
is also investigating whether perceptual enhancements that employ
color, sound, and imagery to emphasize learning-science themes in
video cases can be used to highlight complex thematic interactions
in video cases, helping learners acquire the ability to think flexibly
and interpret classroom interactions from multiple perspectives.
STEP is conducting this research collaboratively with Michigan State
University.
Digital
Insight will provide the critical infrastructure
and process model necessary to allow multisite, multidisciplinary
case building and analysis to become a reality. The components of
the Digital Insight system that will serve the needs of STEP
are:
1.
Web-based support for collaborative viewing, coding, segmenting,
and combining of video;
2.
The capability to enhance video with symbolic overlays and
codes; and
3.
The capability to store and support dynamic assembly and
reassembly of coded, enhanced video segments to produce customized
cases for objectives-based teacher professional development.
Chris
Fassnacht (UW-Madison) - The Interactive Constitution of Children’s
Cognitive Abilities.[11]
Fassnacht will analyze videotaped sessions in which K-6 children
suspected of having cognitive deficiencies are examined using standard
test instruments applied in an interactive setting. Subsequent tapings
of sessions in which the assessments of the children’s cognitive
abilities are delivered to their parents will also be studied. Using
conversation analytic (Sacks & Jefferson, 1992) and ethnomethodological
(Garfinkel, 1967) approaches to these recorded data, Fassnacht will
attempt to uncover some of the collaborative structures by which
objective measures of cognitive capacity are distilled from an interactional
foundation of testing, assessment, professional opinion, and everyday
interaction. One possible outcome of the study is the elucidation
of those structures of interaction whereby individuals are categorized
according to lay and professional knowledge of what constitutes
evidence of normal and abnormal cognitive ability. Digital Insight
will support the digitizing, storage, and management of the video
data collection (approximately 50 hours of tests and meetings).
In addition, Digital Insight will provide access to advanced
software tools for qualitative data analysis and grounded theory
building.
In
each of Years 2 and 3, the Digital Insight project will extend
its tools and support efforts to two additional research teams selected
from a pool of potential candidates, including the TalkBank Classroom
Discourse Working Group (see, e.g., MacWhinney & Snow, 1999)
and other researchers identified by the advisory panel as doing
compelling work supporting high-impact learning in early grades
using video-based research. To date, the following scholars have
indicated their interest in becoming a participating research team:
Rogers Hall (Vanderbilt), Jim Kaput (UMass-Dartmouth), Richard Lesh
(Purdue), Rand Spiro (Michigan State), and Deborah Vandell (UW-Madison).
Project
Deliverables
The
primary deliverables for the project will be both the development
team’s analysis of best practices for collaborative video research
and the substantive work that the new technologies will allow participating
research teams to produce. The ability to intelligently manage and
analyze complex curricula will be greatly enhanced. The ability
to collaborate over distance within a secure environment will both
enable the development of new forms of teaming and address the concerns
of research participants about the security of the data. We will
share results of our own research on best-practice processes and
tools with all participating research teams and the larger academic
community through presentations, applied workshops, and scholarly
writing.
The
Digital Insight proposal is designed to take advantage of
the existence of prototype analytical tools for video analysis and
ongoing efforts to build data exchange and cataloging techniques.
We will be able to incorporate analytical models, markup languages,
and output features directly into our user needs analysis. Our user
communities already recognize the functionality and value of many
of these existing tools.We
will take advantage of work done with commercial qualitative analysis
tools such as NUD*IST, ATLAS.ti, NVivo, and Ethnograph to deliver
an open-coding system that can support a wide range of theoretical
and methodological approaches.
The
Storage Resource Broker and Metadata Catalog developed by the SDSC’s
Data-Intensive Computing Environments team are well adapted to an
audience that is technically sophisticated and engaged in remote
computing. We will also explore the feasibility of adapting the
Metadata Encoding & Transmission Standard (METS) markup schema
for use within Digital Insight. The importance of a standard
cataloging system cannot be overestimated, particularly for large,
long-term projects. The TalkBank proposal addresses the problem
at the individual researcher level through the development of the
ATLAS data exchange standard. Digital Insight will approach
the problem from the institutional level and will include best practices
for the management of complex collections.
One
of the most important - and often overlooked - aspects of data acquisition
is the creation and management of human subject use releases that
adequately describe both what analysis will be done with the data
and how access will be controlled to prevent unacceptable uses.
The Digital Insight team will work with researchers and institutional
review boards and will rely on the expert opinions of the advisory
panel to identify the important dimensions of permission, use, and
repurposing of data. Human subject deliverables will include both
technical solutions to better management of use rights through ensuring
that rights are embedded in the content management system and the
development of consent forms that describe best practices for using
video research data.
Digital
Insight’s integrated user-centered design research
approach will provide ongoing feedback on the development process
and describe changes in research methodologies employed by participating
teams. The advisory panel will support this work at their annual
meetings where they will review the research and evaluation progress,
act as outside reviewers of the development team’s scholarly efforts,
and make suggestions for midcourse corrections in the design and
development efforts. The user-centered design and formative evaluation
deliverables will be tightly integrated to provide data for both
internal feedback and external accountability.
We
will adapt the Digital Insight summative evaluation process
from the CIPP (context, input, process, product) framework originally
developed by Stufflebeam in the early 1970s to evaluate projects
that were not conducive to traditional evaluation and experimental
design methods (Stufflebeam, 1971; Stufflebeam, 2001). The CIPP
framework has been used in a variety of ways over the last 3 decades
to provide ongoing feedback to program implementation, to serve
as a model for systematizing methodological approaches, and to identify
process- and product-oriented indicators of quality (Hekimian, 1984;
Madaus, Scriven, & Stufflebeam, 1983; Metzler & Tjeerdsma,
1998; Seyfried, 1998; Trevisan, 1997). The CIPP framework frees
the evaluation design from assessing the degree to which vague objectives
have been met, allowing the focus to shift to assessing process
and implementation, stimulating planning and changes, and providing
ongoing and useful information for decision making.
Digital
Insight and ROLE
We
recognize that the proposal budget exceeds the amount typically
requested for ROLE funding. It is our belief, however, that ROLE
is the most appropriate program for this work and that broad scope
and scale of the work is vital if one is to test the efficacy and
improve the output of collaborative research. The multi-institutional
and cross-disciplinary research supported within and by Digital
Insight could have a profound impact across much of the EHR
portfolio.
Christopher
Thorn
Thorn
is currently PI on two subgrants from NSF-supported projects. He
is in the 3rd year of a subgrant from the National Partnership
for Advanced Computational Infrastructure (ACI-9619020) to develop
prototype tools for accessing the SDSC-based Storage Resource Broker.
He is also in the 2nd year of a subgrant from the TalkBank
project (BCS-9980009) to participate in the development of an open-source
video analysis tool.
Arcot
Rajasekar
ITR:
Exploring the Environment in Time: Wireless Networks & Real-Time
Management - Award Number 0121726. Start, October
1, 2001 - Expires, November 30, 2004 Total Amount $1,758,079.
KDI:
A Knowledge Network for Biocomplexity: Building and Evaluating a
Metadata-Based Framework for Integrating Heterogeneous Scientific
Data: Award number 9980154; September 15, 1999-July
31, 2002; expected total amount $2,979,000.
Brian
MacWhinney
Tools
for the Linguistic Analysis of the CHILDES Database: Award
number 9808974; September 15, 1998-August 31, 2001; expected total
amount $300,000.
KDI:
TalkBank: A Multimodal Database of Communicative Interaction: Award
number 9980009; September 15, 1999-August 31, 2002; expected total
amount $1,442,000.
