**7. Conclusions**

202 International Perspectives of Distance Learning in Higher Education

Considerable improvement was made on the first question in the post-assessment survey. The students were asked to identify the three Boolean operators. Forty percent of the group answered the question correctly in the pre-assessment test, and 100 percent answered it correctly in the post-assessment test (See Table 3 for a summary of pre- and post-assessment test scores). Eighty percent of the students answered the second question correctly on both the pre-assessment test and the post-assessment test. Question two asked the students to identify the least likely resource for finding citations to articles. The answer was the Texas

Question three asked the students what to look for in determining the authority of an Internet site. Eighty percent answered the question correctly in both the pre-assessment test and the post-assessment test. Question four was challenging for the students. When asked to identify the correct statements in a list that included supposed examples of a book's call number, an ISBN number, a citation to a book, a citation to an article, and a URL address, only 40 percent (two students) answered the question correctly by identifying the correct examples on the list in the pre-assessment test and 60 percent (three students) answered it

Question five asked the students to identify the "word search" that would give them books most directly related to gang violence. One-hundred percent of the students correctly identified "gangs AND violence" as the correct answer in the pre-assessment test and 100 percent also selected the correct answer in the post-assessment test. The results of this and the first question in the test suggest that, by the end of the course, all the students in the group understood what Boolean operators were and how they worked. However, it would also seem that question five is probably too easy and should be replaced by a more difficult

Eighty percent of the students could identify primary research sources in question six in the pre-assessment test, and 100 percent in the post-assessment test. Considerable improvement took place on question seven. The students were asked to identify "typical scholarly research sources" from a list. Twenty percent of the students selected the correct answer in the pre-assessment test, and 80 percent selected the correct answer in the post-assessment

In question eight, 60 percent (three students) in the pre-assessment test and 100 percent in the post-assessment test correctly identified the kinds of information that can be found in the Texas Tech University Libraries' online catalog. All of the students answered question nine correctly in the pre-assessment test, thus indicating that they were aware that full-text magazine articles cannot be found in the catalog. Eighty percent answered this question correctly in the post-assessment test. This indicates regression in learning for this particular bit of knowledge. Question ten asked the students which of two databases—ABI/Inform or Lexis-Nexis Academic Universe—contained full-text newspaper articles. Twenty percent identified the correct answer (Lexis-Nexis Academic Universe) in the pre-assessment test, and 40 percent did so in the post-assessment test. This would indicate that the majority of the students are not aware of the content of this particular database and perhaps the content

All of the students did well on question eleven in both the pre- and post-assessment tests. This question required knowledge of the difference between PDF and HTML full-text

Tech University Libraries' online catalog.

correctly on the post-assessment test.

question.

test.

of databases in general.

This case study examined student learning-outcomes assessment methods that are suitable for Web-based courses. It concentrated on those assessment methods used by instructors at Texas Tech University Library in a one-hour, Web-based credit course developed to teach library research skills to distance students. The instructors selected these methods because they thought that the methods offered the greatest potential for assisting them in reaching the course's assessment goals, included implementation processes that promised to be relatively easy to accomplish, and had attributes that the students would find user-friendly and that would discourage plagiarism. The study also reported the planning process the course instructors follow each year in identifying assessment methods appropriate for their course, in developing and implementing the methods, and how they use the data collected by the assessments to improve the course.

The instructors who teach the library research course participate as team members in the process of continuously improving the course's content and the teaching and learning that occurs in the course. This team effort has evolved over the years into a structured yearly cycle of planning, developing, marketing, implementing, assessing, and improving the course. Among other things, reviewing learning goals, identifying learning outcomes, creating learning activities, enacting learning activities, gathering data to check learning, interpreting data, and enacting decisions occur in this planning cycle. The data collected from the various assessment instruments play an important role in the process because the findings that come from the data help to identify where improvement is needed. Blackboard Learn, the assessment module in Blackboard, will soon be used to track the progress of students enrolled in the online course and will facilitate accurate reporting of the Library's impact on the learning and teaching that takes place at Texas Tech University.

Data collected by the assessment instruments employed in the fall of 2010 disclosed findings that were consulted during the summer of 2011 when plans for improving the learning and teaching experiences that would take place in the fall of 2011 were made. All five students did well on all the quizzes. Scores ranged from six to eight points with a maximum possible

Assessment Methods of Student Learning

examination, a portfolio assignment, or use of a standard test.

questions.

appropriate.

in Web-Based Distance Courses: A Case Study 205

What must the instructors who teach the distance section of LIBR 1100 do to increase still more the learning that takes place in their section? In chapter one of her book *Tools for Teaching*, Barbara Gross Davis maintains that, "in designing or revising a course, faculty must consider what material to teach, how best to teach it, and how to ensure that students are learning what is being taught" (Gross Davis, 2009, p. 3-18). Starting with this introductory statement, she then offers strategies meant to help faculty "make decisions about the content of their course, the structure and sequence of activities and assignments, the identification of learning outcomes, and the selection of instructional resources." The instructors of LIBR 1100's distance section are using Dr. Gross Davis' strategies as one of their aids in developing their course. In addition, they want to continue the process they started in 2008. Each summer, in preparation for teaching in the fall and spring semesters, they plan to meet and agree on what is important for their students to learn. Once they have agreed on what is important, the instructors plan to review the course and, where needed, improve it and bring it up-to-date. During this review, they intend to examine the previous year's assessment data and use the findings revealed by the data to help them decide what needs to be changed. The course's continuous development must include revising all course goals, learning outcome objectives, the course syllabus and schedule, reading assignments, practicums, and quizzes, and writing new materials for added content. Each year, after the course is revised, the instructors need to develop valid assessment instruments that will gauge how well the students are learning what the instructors want them to learn (McMillan, 2001, p. 56-75). The instructors believe that the assessment methods used in 2010 worked well. However, there are other ways to assess including, but not limited to, a final

If the decision is made to continue using pre- and post-assessment tests, future test questions should be determined using a pedagogically sound method, and the instructors need to make sure that the teaching points addressed by all the questions are covered in the course's reading assignments and practicums (Gross Davis, 2009, p. 362-372). In an effort to incorporate active learning into the course, the instructors of LIBR 1100 designed practicums that required the students to use databases, Websites, and other mainly online resources to fulfill the requirements of the assignment (Wexler & Tinto, 2005; Lang, 2008, p. 43-61). These practicums also proved effective in teaching students content. Several of the questions that were answered correctly by more students in the post-assessment test than in the preassessment test assessed specific teaching points the students had learned by doing the practicums. The instructors had previously been concerned about having too many practicums for a one-hour credit course. Perhaps, instead of adding more of them, existing practicums could be expanded to include two or more teaching points addressed in the

Finally, the librarians teaching the distance section of LIBR 1100 must be sure their students have the means to learn the teaching points covered by the test questions (Erickson, Peters, & Strommer, 2006, p. 87-100). One way to do this is through carefully prepared scripts explaining each teaching point addressed in a test question. The scripts could be included among the tools and teaching aids that the instructors refer to during their instructordirected online chat sessions. This practice should assist in reinforcing the learning (Erickson et al., 2006, p. 87-100). Above all, great emphasis should be placed on reviewing the course and its learning-outcome goals every year, and improvements should be made when

score of 8 for eleven of the twelve quizzes. The students' scores for one quiz which had a maximum possible score of five points ranged from four to five points with four of the students receiving five points. Each question on these quizzes tested a teaching point the instructors wanted their students to learn, and the quiz scores indicate that, for the most part, the students learned these points. Automatic grading by Blackboard for the great majority of the questions assured that instructor bias would not affect these scores. Therefore one can assume that the students learned what they were expected to learn from the course reading assignments. The only concern would be that perhaps some of the quizzes might have been too easy for the students, especially quiz one where all the students answered all of the questions correctly.

As mentioned earlier, the six practicum assignments and the annotated bibliography assignment represent authentic assessment of how well students can perform tasks and accomplish projects that are regularly performed by individuals involved in library research. The instructors who developed these assignments felt confident they had created carefully thought out instruments for assessing what they wanted students enrolled in the "Introduction to Library Research" course to learn to perform. The scores on the practicums would indicate that for the most part the students had learned to perform the tasks and projects quite well. With the exception of three, the scores ranged from seventeen to twenty out of a maximum possible score of twenty on all six of the practicums. Seventeen represents 85 percent of twenty. The exceptions were scores of six, ten, and fifteen.

The LIBR 1100 instructors who taught in the fall of 2010 are very satisfied with their students' scores and believe much of what they wanted their students to learn was learned. The one weakness they identified with the way the course was taught was that there was no grading rubric for the instructors to use in grading the practicums, the annotated bibliography, and the handful of fill-in questions on some of the quizzes. Since the Texas Tech University Library instructors take turns teaching the online course, there was concern that these assignments were not being graded consistently. The instructors therefore began developing a grading rubric in the summer of 2011.

The instructors were also rather satisfied with the pre- and post-assessment data. The five students increased their group average score by 18.7 points from pre- to post-assessment. This indicates that they learned several of the teaching points the instructors wanted them to learn. They improved their scores on 8 of the 15 questions in the post-assessment test. In addition, two other questions were answered correctly by all the students in both tests. Six of the questions received the same number of correct answers in both the pre- and post-assessments. Fourteen of the questions in the post-assessment were answered correctly by a majority of the students and one question was answered incorrectly by a majority of the five students. The poor performance on this question indicates that many of the students need to learn more about online databases, and Lexis-Nexis in particular. The way the instructors teach or optimize the learning of databases will be revised for next year. Also, some of the questions were apparently too easy. They should be replaced with more challenging questions. And it appears that greater attention should be placed on teaching the students how to identify numbers and symbols they will run across during their research. These include such things as book call numbers and International Standard Book Numbers (ISBN).

score of 8 for eleven of the twelve quizzes. The students' scores for one quiz which had a maximum possible score of five points ranged from four to five points with four of the students receiving five points. Each question on these quizzes tested a teaching point the instructors wanted their students to learn, and the quiz scores indicate that, for the most part, the students learned these points. Automatic grading by Blackboard for the great majority of the questions assured that instructor bias would not affect these scores. Therefore one can assume that the students learned what they were expected to learn from the course reading assignments. The only concern would be that perhaps some of the quizzes might have been too easy for the students, especially quiz one where all the

As mentioned earlier, the six practicum assignments and the annotated bibliography assignment represent authentic assessment of how well students can perform tasks and accomplish projects that are regularly performed by individuals involved in library research. The instructors who developed these assignments felt confident they had created carefully thought out instruments for assessing what they wanted students enrolled in the "Introduction to Library Research" course to learn to perform. The scores on the practicums would indicate that for the most part the students had learned to perform the tasks and projects quite well. With the exception of three, the scores ranged from seventeen to twenty out of a maximum possible score of twenty on all six of the practicums. Seventeen

represents 85 percent of twenty. The exceptions were scores of six, ten, and fifteen.

The LIBR 1100 instructors who taught in the fall of 2010 are very satisfied with their students' scores and believe much of what they wanted their students to learn was learned. The one weakness they identified with the way the course was taught was that there was no grading rubric for the instructors to use in grading the practicums, the annotated bibliography, and the handful of fill-in questions on some of the quizzes. Since the Texas Tech University Library instructors take turns teaching the online course, there was concern that these assignments were not being graded consistently. The instructors therefore began

The instructors were also rather satisfied with the pre- and post-assessment data. The five students increased their group average score by 18.7 points from pre- to post-assessment. This indicates that they learned several of the teaching points the instructors wanted them to learn. They improved their scores on 8 of the 15 questions in the post-assessment test. In addition, two other questions were answered correctly by all the students in both tests. Six of the questions received the same number of correct answers in both the pre- and post-assessments. Fourteen of the questions in the post-assessment were answered correctly by a majority of the students and one question was answered incorrectly by a majority of the five students. The poor performance on this question indicates that many of the students need to learn more about online databases, and Lexis-Nexis in particular. The way the instructors teach or optimize the learning of databases will be revised for next year. Also, some of the questions were apparently too easy. They should be replaced with more challenging questions. And it appears that greater attention should be placed on teaching the students how to identify numbers and symbols they will run across during their research. These include such things as book call numbers and International

students answered all of the questions correctly.

developing a grading rubric in the summer of 2011.

Standard Book Numbers (ISBN).

What must the instructors who teach the distance section of LIBR 1100 do to increase still more the learning that takes place in their section? In chapter one of her book *Tools for Teaching*, Barbara Gross Davis maintains that, "in designing or revising a course, faculty must consider what material to teach, how best to teach it, and how to ensure that students are learning what is being taught" (Gross Davis, 2009, p. 3-18). Starting with this introductory statement, she then offers strategies meant to help faculty "make decisions about the content of their course, the structure and sequence of activities and assignments, the identification of learning outcomes, and the selection of instructional resources." The instructors of LIBR 1100's distance section are using Dr. Gross Davis' strategies as one of their aids in developing their course. In addition, they want to continue the process they started in 2008. Each summer, in preparation for teaching in the fall and spring semesters, they plan to meet and agree on what is important for their students to learn. Once they have agreed on what is important, the instructors plan to review the course and, where needed, improve it and bring it up-to-date. During this review, they intend to examine the previous year's assessment data and use the findings revealed by the data to help them decide what needs to be changed. The course's continuous development must include revising all course goals, learning outcome objectives, the course syllabus and schedule, reading assignments, practicums, and quizzes, and writing new materials for added content. Each year, after the course is revised, the instructors need to develop valid assessment instruments that will gauge how well the students are learning what the instructors want them to learn (McMillan, 2001, p. 56-75). The instructors believe that the assessment methods used in 2010 worked well. However, there are other ways to assess including, but not limited to, a final examination, a portfolio assignment, or use of a standard test.

If the decision is made to continue using pre- and post-assessment tests, future test questions should be determined using a pedagogically sound method, and the instructors need to make sure that the teaching points addressed by all the questions are covered in the course's reading assignments and practicums (Gross Davis, 2009, p. 362-372). In an effort to incorporate active learning into the course, the instructors of LIBR 1100 designed practicums that required the students to use databases, Websites, and other mainly online resources to fulfill the requirements of the assignment (Wexler & Tinto, 2005; Lang, 2008, p. 43-61). These practicums also proved effective in teaching students content. Several of the questions that were answered correctly by more students in the post-assessment test than in the preassessment test assessed specific teaching points the students had learned by doing the practicums. The instructors had previously been concerned about having too many practicums for a one-hour credit course. Perhaps, instead of adding more of them, existing practicums could be expanded to include two or more teaching points addressed in the questions.

Finally, the librarians teaching the distance section of LIBR 1100 must be sure their students have the means to learn the teaching points covered by the test questions (Erickson, Peters, & Strommer, 2006, p. 87-100). One way to do this is through carefully prepared scripts explaining each teaching point addressed in a test question. The scripts could be included among the tools and teaching aids that the instructors refer to during their instructordirected online chat sessions. This practice should assist in reinforcing the learning (Erickson et al., 2006, p. 87-100). Above all, great emphasis should be placed on reviewing the course and its learning-outcome goals every year, and improvements should be made when appropriate.

Assessment Methods of Student Learning

ISSN 0193-0826

in Web-Based Distance Courses: A Case Study 207

Hufford, J. & Paschel, A. (2010). Pre- and Post-Assessment Tests for the Distance Section of

Ivanitskaya, L., DuFord, S., Craig, M. & Casey, A. (2008). How Does a Pre-assessment of Off-

Kidder, L. (1981). *Selltiz, Wrightsman and Cook's Research Methods in Social Relations* (4th ed.), Holt, Rinehart and Winston, ISBN 978-003-0435-66-9, New York, USA Lang, J. (2008). *On Course: A week-By-Week Guide to Your First Semester of College Teaching*, Harvard University Press, ISBN 978-067-4028-06-7, Cambridge, Mass., USA McMillan, J. (2001). *Classroom Assessment: Principles and Practice for Effective Instruction* (2nd

Mulherrin, E., Kelley, K., Fishman, D. & Orr, G. (2004). Information Literacy and the Distant

Oakleaf, M. (2008). Dangers and Opportunities: A Conceptual Map of Information Literacy

Oakleaf, M. (2009). The Information Literacy Instruction Assessment Cycle: A Guide for

Oakleaf, M. (2011). Are They Learning? Are We? Learning and the Academic Library. *Library Quarterly,* Vol.81, No.1, (January 2011), pp. 61-82, ISSN 0024-2519 Oncu, S. & Cakir, H. (2011). Research in Online Learning Environments: Priorities and

Sima, D., Schmuck, B., Szollosi, S. & Miklos, A. (2007). Intelligent Short Text Assessment in

Su, J., Lin, H., Tsong, S. & Lu, C. (2011). OPASS: An Online Portfolio Assessment and

Texas Tech University, Office of the Provost. (February 21, 2011*).* Core Curriculum

Udo, G., Bagchi, K. & Kirs, P. (2011). Using SERVQUAL to Assess the Quality of E-Learning

United States Department of Education. (2006). *A Test of Leadership: Charting the Future of* 

*Margaret Spellings*, U.S. Department of Education, Washington, D.C., USA

http://www.depts.ttu.edu/provost.councilscmtes/ccc/index.php

*Documentation,* Vol.65, No.4, (2009), pp. 539-560 ISSN 0022-0418

Student: One University's Experience Developing, Delivering, and Maintaining an Online, Required Information Literacy Course. *Internet Reference Services Quarterly,*

Assessment Approaches*. portal: Libraries and the Academy*, Vol.8, No.3, (July 2008),

Increasing Student Learning and Improving Librarian Instructional Skills. *Journal of* 

Methodologies. *Computers & Education*, Vol.57, No.1, (2011), pp.1098-1108, ISSN

eMax, *Proceedings of the 8th IEEE Africon Conference*, Windhoek, Namibia, September

Diagnosis Scheme to Support Web-Based Scientific Inquiry Experiments. *TOJET: The Turkish Online Journal of Educational Technology*, Vol.10, No.2, (April 2011) pp.

Committee Charge and Organization, Texas Tech University Website, August 15,

Experience. *Computers in Human Behavior*, Vol.27, No.3, (May 2011), pp. 1272-1283,

*U.S. Higher Education--a Report of the Commission Appointed by Secretary of Education* 

Vol.50, Nos.5&6, (July, 2010), pp. 139-158, ISSN 0193-0826

ed.), Allyn and Bacon, ISBN 978- 020-5297-51-1, Boston, USA

Vol.9, Nos.1&2, (2004), pp. 21-36, ISSN 1087-5301

pp. 233-253, ISSN 1531-2542

0360-1315

26-28, 2007

151-173, ISSN 1303-6521

2011, Available from:

ISSN 0747-5632

LIBR 1100, Introduction to Library Research. *Journal of Library Administration*,

campus Students' Information Literacy Affect the Effectiveness of Library Instruction? *Journal of Library Administration,* Vol.48, Nos.3&4, (2008), pp. 509-525,
