Preliminary Evaluation Plan for
Biology Workbench Education Enhancements
Dave Stone, University Laboratory High School, Urbana, IL, USA
Introduction
I plan to focus my individual formative project evaluation activities on Biology Workbench Education Enhancements, a computer-based molecular biology project which aims to incorporate molecular biology and computational science into middle school, high school and undergraduate biology teaching. I serve as a Collaborating Educator in this initiative and believe it to be a project which:
Biology Workbench, a molecular biology software tool developed in 1996 by the Computational Biology Group at the National Center for Supercomputing Applications (NCSA) at the University of Illinois at Urbana-Champaign, allows scientists rapid access to protein and genetic sequence databases and sequence analysis tools which reside on numerous remote machines located throughout the world, via a web browser such as Netscape or Microsoft Explorer. Used by researchers in areas ranging from genomic analysis and taxonomy to the physical sciences, its potential applications extend well beyond its current use. Developed to facilitate the work of scientists well familiar with the myriad of established protein and databases, which often differ significantly in communication syntax, Biology Workbench possesses a fairly high learning curve and appears quite foreign to anyone unacquainted with the databases or analysis tools made more accessible through the Biology Workbench interface.
In December, 1997, as part of its charge and in response to recent reports on the need for expansion of technological efforts to achieve the goals of a current education reform, the Education Division of NCSA began efforts toward development of an Education Version of Biology Workbench, presently known as Biology Workbench Education Enhancement. Targeted initially toward high school audiences, current efforts are the result of the combined contributions of University of Illinois scientists, undergraduate and graduate students and a cadre of Collaborating Educators. The team is currently developing an underlying set of tutorials and activities which provide additional support and background for teachers and students using Biology Workbench Education Enhancements. Scaled-down from the full professional version, Biology Workbench Workbench Education Enhancements provides a more limited menu of databases and fewer options for database query, effectively leading the new user to more successful, early use experiences.
My original evaluation plan developed over the past semester is shown below. As result of a meeting last night, it is clear that major administrative changes and the format of project development will change dramatically from the pattern established over the past year. Until Thursday, October 29th I had anticipated the following activity structure for my evaluation plan:
I intend to share my findings with the project's current collaborating constituents (the NCSA Biology Workbench Education Team and Collaborating Educators) as well the students and staff of the Fall, 1998 C&I 490 EIT course.
From the teacher perspective, I believe some aspects of the development are currently in a holding pattern, and have some concerns regarding the future of the project. An article I encountered during Jim Levin's Ed Psych 387 class, Judi Harris' "Organizing and Facilitating Telecollaborative Projects," made clear to me how and why the teacher collaboration component of the project appears to be stagnating. My analysis from July, 1998 can be accessed at http://www.uni.uiuc.edu/~dstone/harris.html .
Project Status Update (Developed Prior to 10/29/98)
Progress regarding each of the eight components of the Biology Workbench Workbench Education Enhancements is outlined below:
1. Assembly and Reading of Pertinent Literature (Completed October 6, 1998).
I began by performing a literature search using the search option on the National Center for Supercomputing Applications home page. The documents I was able to find, or which were recommended by staff associated with the development project, were limited to several NCSA Education Division in-house documents which contained brief paragraphs regarding the initiative and a five paragraph web document, entitled "Education Enhancements to the Biology Workbench," written by Dr. Eric Jakobsson, Lisa Bievenue and Dr. Umesh Thakkar.
2. Individual Interviews Regarding the Evaluation with Dr. Eric Jakobsson (University of Illinois at Urbana-Champaign) and Dr. Umesh Thakkar (National Center for Supercomputing Applications).
Dr Jakobsson has been the Biology Workbench Education Enhancements Project Director and Dr. Thakkar has been the NCSA Education Division Representative in this initiative since its December, 1997 beginning. Both individuals were asked the following questions, though I made an effort to allow them to deviate from the questions as they were presented, steering back to the questions when appropriate.
Summary of Jakobsson and Thakkar Responses and Additional Ancillary Materials Provided During Each Interview
My interview with Dr. Umesh Thakkar took place at University Laboratory High School on October 6, 1998. Dr. Thakkar has been the NCSA Education Division Representative in this initiative since its beginning, and provided some of the historical information presented in the "Biology Workbench Education Enhancements Background" section above. Thakkar believes the major stakeholders to be 1) the Education Division of NCSA, 2) the local teachers involved in development, 3) a group called NCSA Alliance Partners, 3) U of I scientists associated with Biology Workbench Education Enhancements development and 4) BioQuest, a commercial education product developer which has an interest in incorporating some (all?) aspects of Biology Workbench Education Enhancements Background into its development efforts. Timelines for project completion were established in NCSA in-house documents, and Thakkar stated that those timelines have been met. Project funding appears to have come almost exclusively from the Education Division of NCSA. A recently awarded Education, Outreach and Training grant pays the salaries of two undergraduate students who are working on the project. Thakkar believes the strengths of this project include the involvement of middle and high school teachers, equitable sharing of work between all of the parties involved in development and the provision of technology (retired NCSA Macintosh 8100s) and a $100 stipend for each of the teachers. Rather than dealing with weaknesses, Thakkar went on to discuss the next step in the project, which should include expansion of topics beyond the current four topics in development at this point, more presentations by teachers, funding for continued teacher involvement and expansion to include a number of additional teacher-developers. Thakkar also chose to address aspirations when queried regarding areas of concern. His main aspiration was that there be continued dialogue between the development team, teachers, undergraduates and U of I faculty. He suggested that the major area of evaluation efforts should center around web design. Specifically, can teachers successfully navigate and use the existing interface? He suggested that I pick one of the two current lessons I had initially suggested (either sickle cell anemia or cystic fibrosis) and examine it in depth. At this point, neither version of Biology Workbench has undergone a formal evaluation, though one of Thakkar's Summer, 1998 activities involved examination of how practicing molecular biologists use the current version of Biology Workbench, the professional version.Three practicing molecular biologists, considered experts in using Biology Workbench, were observed as they used this tool during May 31 - June 4, 1998. Two of the three scientists had difficulty using the query functions and understanding its required syntax. One scientist used Biology Workbench for initial searches and then used those results to continue her efforts using a different database she considered more robust. Participants liked the design of the site and the interface, though they found completion of tasks using Biology Workbench somewhat frustrating.
Dr. Eric Jakobsson, Biology Workbench Education Enhancements Project Director , was contacted on Tuesday, October 6, 1998 electronically regarding the possibility of a meeting.
3. Examination of Web Site For Organization, Readability and Interface Considerations (Completed October 17, 1998)
I began this component of the evaluation by using six of the more commonly used Internet search engines: AltaVista, Excite, Hotbot, Infoseek, LookSmart, and Lycos. Interestingly, two of the search engines (AltaVista and Looksmart) found several of the Biology Workbench Workbench Education Enhancements pages, two (Excite and Infoseek) found only the project home page and two (Hotbot and Lycos) were unable to find any pages associated with the project.
The project home page features an attractive logo and three sentence fragments which I assume to be placeholders for information yet to be stated. As currently presented - "Teacher resources. Tutorials using the Biology Workbench. to help students learn the science of biology, bioinformatics, evolution, and more." - this place holding section looks unprofessional, sloppy, giving the reader a poor initial impression of the project and likely leaves them wondering about the degree of English literacy possessed by the project developers. The project home page currently has links to five sections: 1) Introduction to Biology Workbench, 2) Tutorials, 3) Collaborating Educators, 4) NCSA Development Team and 5) Templates for Creating Tutorials. "Introduction to Biology Workbench," an introduction to the professional version, is well written and organized. Developers should consider adding a second introduction, entitled "Introduction to Biology Workbench Education Enhancements," that deals specifically with the education version and includes a brief history and presentation of this version's goals and objectives. The "Tutorials" page is awkward in its appearance, and could be streamlined, with the topics divided into two major categories: Molecular Evolution Tutorials and Genetic Disorder Tutorials. The contributor pages ("Collaborating Educators" and "NCSA Development Team") differ dramatically in their appearance. The "NCSA Development Team" page is nicely formatted, though I suggest the major players, Jakobsson and Thakkar, be listed first and the rest of the development team be listed in alphabetical order. The "Collaborating Educators" page should use the presentation format of the "NCSA Development Team" page. Collaborating Educators should be listed alphabetically, with links to their personal and school home pages. Listing of the grades and disciplines taught is not necessary, though those can be included. The final section, Templates for Creating Tutorials, is clearly in a (pre-?) beta testing stage, so I won't deal with it at this point, though I'm very pleased with what I see of the template so far, and think once it is developed, it will significantly increase the usefulness of Biology Workbench Workbench Education Enhancements incorporation into specific classroom lessons, and may allow for successful use of this application for some activities at the middle school level. In summary, I suggest the development team consider expanding the number of sections and modifying their order to the following: 1) Introduction to Biology Workbench, 2) Introduction to Biology Workbench Education Enhancements, 3) NCSA Development Team, 4) Collaborating Educators, 5) Tutorials and 6) Templates for Creating Tutorials.
In terms of initial appearance, the site is well designed, with an attractive logo at the top of each page. I suggest the page designers consider a smaller version of the logo to reduce, and in some cases eliminate, the need for scrolling. A counter indicating the number of visitors is included, dates of most recent update, and consistently positioned Home buttons allow for rapid navigation throughout the set of documents. The lack of Java-based items allows for rapid uploading of all documents.
4. Use of Tools Through Two Activities (Ajit's and Paul's)
I had planned to perform the following activities in my classroom during mid-November, 1998, incorporating analysis of each of these activities into my evaluation:
Biology Workbench Education Version Tutorial (2 day activity) http://www.uni.uiuc.edu/~dstone/lesson2.html
Human Evolution Activity Using Biology Workbench (2 day activity) http://www.uni.uiuc.edu/~dstone/lesson3.html
As will be clear from the section entitled "A New Direction for Biology Workbench Education Enhancements", which appears later in the document, this timeline has been modified due to a new change in project approach and working group structure, announced on October 28, 1998. See that section for specific changes.
5. Development of an Evaluation Instrument for Students Using the Activity
I had planned to develop an evaluation instrument using Likert-type questions selected from the University of Illinois ICES questions. Additionally, I planned to include several open-ended, free response-type questions. I mentioned the evaluation questions to Dr. Umesh Thakkar during our October 6 th meeting, and we tentatively set a meeting time (Tuesday, October 27, 1998) for us to discuss the evaluation questions that I selected. Due to Biology Workbench-related activities, he was unable to meet. Our meeting was indefinitiely postponed.
6. Student Use of Activity
Initially I anticipated having my Field Biology students spend four days during mid-November, 1998, working with Biology Workbench Education Version Tutorial and Human Evolution Activity Using Biology Workbench, though the new change in project approach and working group structure, outlined in "A New Direction for Biology Workbench Education Enhancements" explains why I will need to modify my originally planned activity student group and timeline.
7. Student Evaluation (including both qualitative and quantitative components)
I will develop a questionnaire (see #5 above) for the students involved in this developemnt project, though format and timeline have yet to be established.
8. Use of Activity Construction Template for Actual Activity Development
Assuming that this template is at a working stage prior to the due date of this project, I will work through the template to develop an evolution-based molecular biology demonstration, and then critique the template and its instructions, as well as include suggestions for further modification.
A New Direction for Biology
Workbench Education Enhancements (October 29, 1998)...
The first meeting of all involved parties (NCSA Biology Workbench Education Team and Collaborating Educators) to be held in five months occurred on October 28, 1998. It was clear from the onset, that the entire Biology Workbench Education Enhancements effort is currently undergoing a major restructuring in terms of personnel, development approach and audience. A flurry of web site activity had occurred over the past few days, with a number of minor changes in the interface and major restructuring of some of the tutorials and lessons, including more screen hots and explicit directions. It appears that a major change in the project PIs has occurred, with Umesh Thakkar's original role being reassigned to Raul Zaritsky and Scott Lathrop of NCSA's education division, though none of this information was made official at the meeting. A number of new modules including an evolutionary molecular biology and a developmental protein module were presented. Zaritsky suggested a new structure in which teachers and developers work together in small groups on development of teacher-initiated modules and perform several rapid trials with ever-increasing numbers of students. I think that the change in structure is appropriate and will likely bring us back on track. A new module dealing with glycolysis/fermentation/respiration is currently being discussed and will likely be written during the academic year. The NCSA Biology Workbench Education Team has been working with a BioQuest, a commercial software education company. Their discussions will likely shape the future direction of Biology Workbench Education Enhancements efforts. It appears that the development focus is moving from middle and high school to a much increased focus on undergraduate level activities, centered particularly within the area of biochemistry.
... And Therefore a New Structure for My Individual Project
As a result of the meeting, I have changed my evaluation plans to some degree. I will be working with Ajit Chary, an undergraduate biology major who is a member of the NCSA Biology Workbench Education Team, on his molecular evolution activity. He intends to put a new version of his activity on the web at some point next week, after which I will perform and critique the activity. This will be followed by Ajit making my suggested modifications to the activity. Finally, Ajit and I will work with a group of my students on the activity. I like the new approach to collaboration and development and think the revised approach will significantly increase the likelihood of production of a quality, easily disseminated product.
I'm somewhat disappointed that my evaluation of Biology Workbench Education Enhancements will not be completed to the degree I had initially anticipated prior to the due date for this project. On the other hand, I'm quite pleased with my selection of this project as my area of focus, since I'm even more certain now that my evaluation will play an important role as an historical document and it will significantly contribute to the role that I play in this development effort.
Tentative Final Evaluation Report Outline
I. Introduction to Biology Workbench Education Enhancements
1. Purpose of the Evaluation (will incorporate part of the "Introduction"
)
2. Audience of the Evaluation Report (will incorporate "Evaluation
Audience")
3. Limitations of the Evaluation and Explanation of Disclaimers (will incorporate
components of "A New Direction for Biology Workbench
Education Enhancements...")
4. Overview of the Report Contents
II. Focus of the Evaluation
1. Description of Biology Workbench Education Enhancements [will incorporate
components of "Biology Workbench (Professional
Version) Background" and "Biology Workbench Education Enhancements
Background"), as well as interview information)]
2. Questions/Objectives Used in This Study (will incorporate interview questions,
standards/objectives Ajit and I use in my evaluation of
modules and student feedback regarding activities performed as time permits)
III. Overview of Evaluation Plan and Procedures
(will consist of a modified version of "Evaluation Plan", incorporating
components of "... And Therefore a New Structure for My Individual
Project.")
IV. Presentation of Evaluation Findings
1. Summary of Evaluation Findings
2. Interpretation of Evaluation Findings
V. Conclusions and Recommendations
1. Criteria and Standards Used to Judge the Evaluation Project (will
incorporate standards outlined in "Examination of Web Site For
Organization, Readability and Interface Considerations")
2. Judgments About the Evaluation Project - Strengths and Weaknesses (will
include discussion of strengths and weaknesses of the
project as well as Ajit's module)
3. Recommendations
Biology Workbench clearly has excellent potential to be used in middle school, high school and undergraduate teaching situations. I have several suggestions regarding 1) molecules to consider, which will better assist other teachers in fully understanding the potential of Biology Workbench, 2) introductory topics to consider in middle and high school module development, and 3) advanced topics better suited to students with previous Biology Workbench experience or currently taking undergraduate biology courses. I hope the following ideas will serve as a catalyst for further conversation and project emphasis consideration.
1. Comparison Molecules to Incorporate Into Tutorials
2. 1st and 2nd Year Biology Topics Topics Workbench Can Reinforce
3. 2nd Year High School/Undergraduate Biology ConceptsTopics Workbench Can Reinforce
Finally, a number of generally useful visualizations could easily be
combined into relatively easy tutorials, which would be broadly useful for
students regardless of grade level. I suggest the development team begin
with protein structure and mutagen impact visualizations. The protein visualization
should allow the user the ability to select and/or make up a nucletide or
coding base sequence, seamlessly examine its secondary and tertiary structure
and see how the polypeptide chain might interact with other chains (quaternary
structure). The mutagen impact visualiation should allow the user to select
a base sequence, induce a mutation in a "hot spot" (repetitive
sequence) by exposing it to various different mutagens (e.g. uv light, asbestos,
etc). It would be great if the visualization would allow each user to have
a split screen, showing the normal and mutated proteins, as well as allow
the user the ability to rotate each of those molecules simultaneously. This
would nicely reinforce the idea that mutation impacts can range from negligible
to severe.
Created October 3, 1998. Last modified November 5, 1998.