Assignment #2: Major Project
Marissa Galindo Alcorta
IRLS 501
Fall 2004
Evaluation of Metadata Quality for Educational Resources

Resources from:
Digital Library for Earth System Education
Resource Subject:
History and Philosophy of Science
Metadata Evaluation Criteria (Coleman):
Consistency
Accuracy
Granularity
Level of Subject Indexing


Part I. Evaluation of Metadata for Educational Resources (Using Four Listed Criteria)
   
    Taking the various criteria for evaluating metadata quality as a template I reviewed ten electronic resources (websites/pages) from DLESE, the Digital Library for Earth System Education. This online digital library contains a listing of  educational based online resources that focus on various areas of Earth based Science, such as astronomy, geology, paleontology, and climatology, among others. My subject of focus in my search was History and Philosophy of Science pertaining to earth systems. The descriptive records contained in my chosen  DLESE electronic resources were basically created in a DC (with IEEE LOM and ADN), therefore this was the metadata framework that was examined in my criteria evaluation (Coleman, deCharon, et al.).
     Educators and students that utilize online resources often struggle with determining the validity of electronic resources in their learning processes. Utilizing online resources are fast becoming a common trend in classroom environments through the increase of multimedia materials. Educators and students have limited learning time to search aimlessly through search engines, databases, and digital libraries for materials that would benefit them in their educational endeavors. By looking at and evaluating online metadata records using quality criteria, such as the above mentioned, educators and students can better evaluate the resources they come across as well as create a more time efficient learning environment for themselves and their students.
    Four initial criteria were used to evaluate my chosen metatdata records in the course of the project. Educators and students can use these criteria as a basis for evaluation of metadata records and other online resources and as a tool to better choose electronic resources that are appropriate to their classroom learning environment. With this basis in mind and as a student user of the DLESE digital library, I utilized the aforementioned criteria to evaluate the educational quality of ten chosen electronic resources and their metadata records. These four criteria, which are consistency, accuracy, granularity, and level of subject indexing, are all crucial to metadata evaluation in terms of educational relevance and appropriateness (Coleman).
    These four evaluation criteria are all important tools in metadata quality control  for the educational resources. Consistency (how consistently names and terms are used in describing the resource), accuracy (do the resource description match the metadata record), granularity (the depth and level in which a resource is described), and the level of subject indexing (exhaustive or summative) are all important criteria educator and students can utilize and employ to make more informed choices in the electronic resource use (Coleman).
    The first electronic resource and metadata record I evaluated using the four determined criteria was the Arthur Ross Hall of Meteorites (see Appendix 1). The metadata record for this resource did have consistency in it's word and name use, leading the user to know that the resource pertains to meteorites. There was accuracy between the record and the resources, in that the titles and other components did meld. In terms of granularity, the record did describes the various lower levels and links within the website. In terms of the level of subject indexing, the subject headings and keywords used in the record were very straightforward and summative in description.
    My second evaluated electronic resource and metadata record was ECHO: Exploring and Collecting History Online, which dealt with the collection of history in the realm of science and technology (see Appendix 1). The record for this resource was consistent in that various key names/terms were used throughout the description. The accuracy of the record was less obvious in that the title on the metadata record did not match exactly with the resource itself. The metadata record left out the ECHO acronym, which may cause problems in keyword searching by a user. The granularity of the record was deep in that it described the various component (links) embedded in the resource. Once again the level of subject heading was summative, listing only two keywords in the subject heading.
    The third resource and metadata record was Students for the Exploration and Development of Space: The Messier Catalog (see Appendix 1). The record did consistently use the same names and key terms throughout the description. There was some inaccuracy between the metadata record and the actual resource in that the web page used an acronym as the title (SEDS) and the metadata record used the above phrase as the title, an inaccuracy which may lead to some confusion in terms of use. The granularity of the record was very developed and had depth in that there was description available of the various levels of the site. In terms of subject indexing, like the others, the level was summative, with only two key words/phrases used as subject headings.
    My fourth chosen resource and metadata record was Faces of the Earth Class Page, which is a home page for a freshman Earth Sciences course at the University of Minnesota (see Appendix 1) . In looking at the record one could say that there was consistency in the various terms and names used throughout the record. The accuracy between the title on the metadata record and the actual sources was off in that the metadata record did not list the course number (GEO 1901-001) in the title and the actual resource did. The granularity of the record was top level in that none of the various links and components of the resource were described in the record. The level of subject indexing within the metadata record was somewhat more descriptive of the resource but, overall, summative in the use of subject keywords/terms.
    The fifth resource/record I evaluated was The Evolution Evidence Page, which pertained to the topic of evolution and the various debates, issues that surround this area of study (see Appendix 1). In terms of the consistency of the overall metadata record, key terms, such as evolution, were used throughout the descriptive properties of record. There was inaccuracy, once again, in the titles of the record and resources site. The actual resource site did not have any title listed, which may lead a user of the site to uncertainty about accessing the correct site. The granularity of the metadata record was top level, and did not show any evidence of the various links and embedded pages within the electronic resource. The level of subject indexing in the record was summative and listed only two keywords as subject headings for the resource as a whole.
    My sixth evaluated resource/record was Amazing Earth: Facts that Fascinate (see Appendix 1). The purpose of this resource was to provide an internet activity where students would use search engines to find "facts" on the subject of Earth Science. The consistency of names and terms used throughout the metadata record was very low in that I only found the "earth science" keyword mentioned once within the record. The accuracy between the record and the actual resource was also low in that the description on the record overemphasized and seemed to exaggerate what the actual resource was, a link to a basic search engine. Granularity was top level simply because there were not levels or embedded components in the resource to describe. Finally, the level of subject indexing the record was again summative as only one phrase was used as a subject heading. Overall, the metadata record lack depth and had little description due to the lack of educational value the resource itself possessed.
    The seventh resource in my evaluation of the metadata record was the CERES SCOOL Project (see Appendix 1), which pertained to a class project on the CERES program ( a NASA satellite program that measured cloud energy). The metadata record for the resource was consistent in it's use of key terms and names, such as atmospheric science, clouds, and remote sensing. There was accuracy between the title listed on the metadata record and the resource site. Granularity of the record was at a somewhat lower depth in that other levels (links) embedded in the site were acknowledged, such as the alternative sites in Spanish, French, etc.   In terms of subject indexing, the level was more exhaustive than previous site in that the subject headings and various components of the overall website were indexed in a highly  descriptive manner.
    My eighth chosen resource/record was Exploratorium: Science Fairs, which was an educational resource on questions and suggestions on student science projects (see Appendix 1). The metadata record for this resource was consistent in describing the main focus of the resource, science fairs and projects. The accuracy of the site was very high in that the title headings of both the resource and record matched and the overall description in the record was accurately reflective of the resource. This record did have depth in it's granularity level in that the various level/components embedded within the resource were acknowledged and described in the metadata. The level of subject indexing in the metadata record for the resource was summative and contained only two subject heading phrases as descriptive.
    The ninth evaluated resource/record was Windows to the Universe, which was an educational resource that pertained to bringing the arts and humanities into the field of earth science (see Appendix 1). The overall consistency was very high for this record in that key names and terms were used not only within the description but also throughout the various elements of the record. The accuracy between the metadata and the resource was high and various components, such as the tile headings, did match up between the two. Granularity was in the lower levels, with quite a bit of depth in the metadata description of the resource and also in the various embedded links the resource contained. The level of subject indexing for this metadata record what exhaustive in that it was highly descriptive and use a large amount of subject headings terms to describe the resource.
    My last evaluated electronic resource and metadata record was Fossils and Myths: Ancient Paleontology, which pertained to the science of paleontology and the various historical myths that surround this issue (see Appendix 1). The metadata for this resource was somewhat inconsistent in that there was little uniformity between the phrases used in the descriptions (Greeks, mythical beasts, planet Earth) and the names terms used in the rest of the record (paleontology, geological sciences). The was little accuracy between the record and the resource in that the titles were different. The resource itself was titles Pulse of the Planet, completely different from the metadata title. Another inconsistency was that the metadata record describes the resource as a "two-minute sound segment" and the actual resource is text-driven. In terms of granularity, the descriptive level was top level, with none of the embedded links described in the metadata. Finally, the subject level of the metadata was summative and had only two descriptive terms under as subject/keyword headings.


Checklist for Evaluation of Metadata= 4 Criteria (Coleman)
Resource Title and URL
 Consistency
(High or Low)
 Accuracy
(High or Low)
 Granularity
(Top Level or Lower Level)
 Level of Subject Indexing
(Exhaustive or Summative)
1. Arthur Ross Hall of Meteorites
http://www.amnh.org/exhibitions/permanent/meteorites/what/whatis.php
 High
 High
 Lower Level
 Summative
2. ECHO: Exploring and Collecting History Online
http://echo.gmu.edu/
 High
 Low
 Lower Level
 Summative
3. Students for the Exploration and Development of Space: The Messier Catalog
http://www.seds.org/messier/index.html
 High
 Low
 Lower Level
 Summative
4. Faces of the Earth Class Page
http://talc.geo.umn.edu/courses/1901-1/
 High
 Low
 Top Level
 Summative
5. The Evolution Evidence Page
http://www.gate.net/~rwms/EvoEvidence.html
 High
 Low
 Top Level
 Summative
6. Amazing Earth: Facts that Fascinate
http://www.chemsoc.org/networks/learnnet/jesei/fascinat/students.htm
 Low
 Low
 Top Level
 Summative
7. The CERES SCOOL Project
http://asd-www.larc.nasa.gov/SCOOL/
 High
 High
 Lower Level
 Exhaustive
8. Exploratorium: Science Fairs
http://www.exploratorium.edu/ls/pathfinders/scifairs/index.html
 High
 High
 Lower Level
 Summative
9. Windows to the Universe
http://www.windows.ucar.edu/
 High
 High
 Lower Level
 Exhaustive
10. Fossils and Myths: Ancient Paleontology
http://www.pulseplanet.com/archive/Apr03/2895.html
 Low
 Low
 Top Level Summative


Part II. Literature Review and Evaluation of Metadata Using Developed Criteria
  
     In working to develop other metadata evaluation criteria for educational purposes I wanted to focus on what I as a former teacher would find useful when searching for resources in my teaching. Beyond my using my own personal experiences I also utilized the available literature pertaining to cataloging, classification, and metadata creation as a basis for creation of new metadata evaluation criteria. The process of developing new criteria is important in that utilizing the metadata record for an electronic resource is an extremely useful tool in that it provides an overall snapshot of a resource and allows the educator/student to quickly evaluate if a website/page will be of use to them and meets their educational needs. This idea of a "snapshot" evaluation of a electronic resource becomes extremely crucial to educators and students who have limited time and resources to do extremely extensive and undirected searching online.
    As a former educator and as a present student I can appreciate the potential effectiveness metadata-based searching can pose in any learning environment. Yet, as has been demonstrated by my previous evaluations there is much that has to be improved in terms of metadata creation. Above all, the overall descriptiveness of the metadata record must be improved, with all aspects and levels (granularity and subject indexing) of the electronic resource described with much more specificity. This is crucial so educators and students can utilize metadata and be completely clear on what subject the electronic resource pertains to and what educational value and benefit the various resource components will have on their educational goals. By developing this specificity, educators, students, and the general public can better
decipher metadata and use this resource to their educational advantage.
    Yet, metadata creation has not yet reached this level of  descriptive specificity and many educators/students outside the Information Science see no value in metadata searching. I believe that by creating criteria that is useful within the fields of education (teaching and learning) and is based on some attributes many teachers and students look for in electronic resources these groups will begin to see the value metadata has in their world. In creating new evaluation criteria, it is imperative that  we use the educators and student that will eventually use this tool as the template for any success. Keeping the ultimate goals of a more successful teaching and learning environment in mind I developed four other criteria for metadata evaluation of educational resources. They consist of Language, Age-Appropriate Content, Intellectual Accuracy, and the Country of Origin of the resource.
    Language plays a crucial part in all classroom environments in the United States today. As the U.S. is continues to become an increasingly multicultural and multilingual population educators will begin to see this trend reflected within their classrooms and learning environments. Educators must adjust to this diverse student population and provide the necessary resources, in various languages, for their students to learn and intellectually grow more effectively. Therefore, it is imperative that electronic resources used in a learning environment are multilingual, if possible. Beyond this, it is important, in terms of time efficiency, that some language component is shown within the metadata record so educators and students can evaluate if this an electronic resource that will meet their language needs (Merrill).
    A second criteria I developed for metadata evaluation of educational resources plays off  the criteria movies and television are rated which I will define as Age-Appropriate Content. By utilzing electronic resources on the internet within a classroom setting educator and student are exposing themselves to various degrees of content (sexuality, violence, etc.) that may be inappropriate for their students age group and a classroom-setting as a whole. It is imperative that educators are able to evaluate what type of content electronic resources contain and if the content is at the level at which they wish to involve their student body. By creating a rating system within the metadata record based on the G, PG, PG-13, and R ratings used for movies/television educators will be able to make more informed decisions about educational electronic resources. While I am not calling for censorship of resource content I feel it is important that educators have tools to be more informed when it comes to utilizing electronic resource (Milstead and Feldman).
    My third developed criteria for evaluation of metadata in electronic resources is Intellectual Accuracy. In looking at educational resources and their metadata record for the DLESE digital library I was accutely aware of  the importance of providing the users of this resource with accurate content. This intellectual accuracy becomes extremely important when the electronic resources are based on scientific research, methods, and discovery. New information is created in the scientific world on a daily basis and for the electronic resources in this topic to be of any intellectual use to educators and student users they must be updated on a regular basis .In order for educators to keep track of this accuracy, any information updates should be reflected in the metadata record in terms of the most recent dates they occur. If these updates do not occur then the scientific-based information contained within can become intellectually inaccurate and obsolete and of little educational use in terms of learning and teaching (Graham).
    The final criteria I developed in order to better evaluate the metadata record for educational electronic resources looks at the Country of Origin of the resource. Indication in the metadata record of a country of origin for the material found in an electronic resource is extremely important for educators to understand the cultural, social, and political background of the information contained within. This background is especially useful in terms of evaluating scientific-based electronic resources in that scientific theory, methods, and practices differ between countries. By indicating the national background of the resource within the metadata record educators can get a quick view of  where the theory behind the scientific information they are viewing originated from and can better adjust and utilize the resource to their educational needs. Beyond this, listing a country of origin in the metadata record would also be useful for educators who wish to use electronic resources that have a regional connection to their teaching institutions and place of student residence. For example, many educators may seek out electronic resources that were created and have ties to their communities and cities so their students may feel a greater connection to the educational resource material being presented (Merrill).
   Taking my developed evaluation criteria for metadata quality I again looked at my ten chosen resources to evaluate how useable the criteria was in terms of how the metadata is presented currently. Needless to say, the metadata record for my electronic resources presently do not contain any of the criteria that I developed. It was very difficult to to determine three of the new criteria I developed (Age-Appropriate Content, Intellectual Accuracy, and Country of Origin) simply because there was no DC element present that showed this criteria in the metadata record. Trying the determine Age-Appropriateness for my metadata record/resource was virtually impossible without some uniformed fixed criteria embedded in the record that measured this criteria. Intellectual Accuracy of  the resource was also difficult to determine within the metadata record due to the records having little date information. In Resources #2 and#10, I gleamed some date information from the copyright dates provided in these two metadata records. But, this date did not reflect any updates that may have taken place after the creation date. The Country of Origin of my resources was also difficult to pinpoint from the metadata records. In Resources #1, #2, #3, and #4 I gleamed the country of origin of the resource by looking at the educational affiliations of the sites, many of which were American universities.
     Finally, of all the criteria I developed Language was the only one that had some mention within the present set-up of the metadata record. Resources #3 and #7 actually listed the various languages the resources was available in the descriptive paragraph of the metadata record. So, although language was not listed as a DC element the record did mention this criteria. In terms of the the remaining resources (#1-2, 4-6, and 8-10), I made the assumption that these resources were only in English by looking at the language contained within the metadata record.
    While I did come across some difficulty evaluating the metadata record of my chosen resources with my developed evaluation criteria one can see that these problems arose from the lack of uniformity and consistency in the metadata record as it is available now. Furthermore, these evaluation problems also arose from the fact that these criteria I developed are new and therefore not even available within the metadata record. Stemming from this inconsistency, I believe uniformity and user-friendliness in metadata creation should be a goal for the future of education and information science. Ultimately, metadata creation needs to move towards a trend where the informational needs of users (educators/students) are taken into account and included within the metadata record. Only with this specificity will more segments utilize the little known resource of metadata (Coleman, deCharon, et al.).
    By developing these additional criteria I added to the list of criteria already available for metadata evaluation educators, students, and the general public so they can better utilize this resource that, trend-wise, has not be used consistently outside the sphere of information and library science. These criteria would be extremely useful to educators and students who are seeking intellectually accurate, age-appropriate, and culturallly/socially relevant electronic resources to add to their teaching and learning environments. By creating education-relevant criteria for metadata and resource evaluation perhaps many digital libraries, such as DLESE, will see this as a step in creating a consistent set of metadata elements that would be user-friendly for all segments of society.
 
   
Part III. Developed Checklist-Metadata Quality for Educational Tasks
Resource Name
Url:
 Language




(Merrill)
 Age-Appropriate Content
(G, PG, PG-13, R)


(Milstead/Feldman)
 Intellectual Accuracy ((date most recently updated)


(Graham)
 Country of Origin



(Merrill)
1. Arthur  Ross Hall of Meteorites http://www.amnh.org/exhibitions/permanent/meteorites/what/whatis.php English
 Unknown
 Unknown
 United States
2. ECHO: Exploring and Collecting History Online
http://echo.gmu.edu/
 English Unknown
 Unknown United States
3. Students for the Exploration and Development of Space: The Messier Catalog
http://www.seds.org/messier/index.html
 English, German, Italian, and French
 Unknown
 1995-1997 United States
4. Faces of the Earth Class Page
http://talc.geo.umn.edu/courses/1901-1/
 English
 Unknown Unknown
 United States
5. The Evolution Evidence Page
http://www.gate.net/~rwms/EvoEvidence.html
 English
 Unknown Unknown Unknown
6. Amazing Earth: Facts that Fascinate
http://www.chemsoc.org/networks/learnnet/jesei/fascinat/students.htm 		English Unknown Unknown Unknown
7. The CERES SCOOL Project
http://asd-www.larc.nasa.gov/SCOOL/ 		English, Spanish, French, Italian, German, and Chinese
 Unknown Unknown
 United States
8. Exploratorium: Science Fairs
http://www.exploratorium.edu/ls/pathfinders/scifairs/index.html 		English
 Unknown Unknown
 Unknown
9. Windows to the Universe
http://www.windows.ucar.edu/ 		English
 Unknown Unknown
 Unknown
10. Fossils and Myths: Ancient Paleontology
http://www.pulseplanet.com/archive/Apr03/2895.html 		English
 Unknown 2002
 Unknown


Appendix 1
Resource Title
 URL
1. Arthur Ross Hall of Meteorites
 http://www.amnh.org/exhibitions/permanent/meteorites/what/whatis.php
2. ECHO: Exploring and Collecting History Online
 http://echo.gmu.edu/
3. Students for the Exploration and Development of Space: The Messier Catalog
 http://www.seds.org/messier/index.html
4. Faces of the Earth Class Page
 http://talc.geo.umn.edu/courses/1901-1/
5. The Evolution Evidence Page
 http://www.gate.net/~rwms/EvoEvidence.html
6. Amazing Earth: Facts that Fascinate
 http://www.chemsoc.org/networks/learnnet/jesei/fascinat/students.htm
7. The CERES SCOOL Project
 http://asd-www.larc.nasa.gov/SCOOL/
8. Exploratorium: Science Fairs
 http://www.exploratorium.edu/ls/pathfinders/scifairs/index.html
9. Windows to the Universe
 http://www.windows.ucar.edu/
10. Fossils and Myths: Ancient Paleontology
 http://www.pulseplanet.com/archive/Apr03/2895.html


References

Coleman, Anita.(2004) Guide to Selecting and Cataloging Quality WWW Resources for the Small Library. Learning Resources Association of the California             Community Colleges. Fairfield, California.

Coleman, A., deCharon, A., Frost, C.O., Ginger, K., and Raskin, R. (2004). A Framework for the Future of Educational Digtial Libraries: Metadata and                Vocabularies for Learning.

Digital Library for Earth System Education (DLESE). http: //www.gpoaccess.gov/cgp/

Graham, P.S. (1990). Quality in Cataloging: Making distinctions. The Journal of Academic Librarianship, 16(4), 213-218.

Merrill, W.S. (1912). A Code for Classifiers: Its Scope and Its Problems

Milstead, J., and Feldman, S. (1999). Metadata: Cataloging by Any Other Name. Online, Inc.

Taylor, A.G. (2004). The Organization of Information (Second Edition), Westport, Connecticutt. Libraries Unlimited.

KR Toolbox