**2. Literature review**

Searches in several bibliographic resources using the keywords "higher education," "distance learning" or "distance education," and "assessment" yield a large number of articles, books, documents, and other materials on assessment. Some of these materials are guides, manuals, or action plans; articles on the need to integrate assessment into education; or reports of accreditation trends in higher education. Other materials found discuss strategies used to gain support for or to develop assessment programs, or report on statewide assessment programs of higher education curricula without the details of any particular assessment projects. The author found a less numerous group of articles that in some way or another discuss assessment of student learning in distance courses, including those that report the actual experiences of distance learning instructors in assessing student learning in their courses. Though a few of these articles report on studies that did not take place in a higher education environment, higher education faculty involved in distance learning should be aware of these studies because of their pertinence in the developing field of distance learning in higher education.

<sup>3</sup> Web-based learning (also called online learning, e-learning, or electronic learning) comprises all forms of electronically supported learning and teaching. The information and communication systems, whether networked or not, serve as specific media to implement the learning process.

This chapter is in the form of a case study. It examines student learning-outcomes assessment methods that are suitable for Web-based courses, and points out some of their strengths and weaknesses as seen from the perspective of the online learning environment. The author concentrates on those assessment methods used by instructors at Texas Tech University Library in a one-hour, Web-based credit course developed to teach research skills to distance students.3 After reviewing the various categories of methods available, the course instructors decided to use those that they thought offered the greatest potential for assisting them in reaching the course's assessment goals, comprised implementation processes that promised to be relatively easy to use and not so time-consuming, and included attributes that would facilitate student use. The course instructors' method for using assessment data to improve their course is demonstrated by way of examining the data collected in the fall of 2010 to determine findings that could help identify problems that need to be fixed in order for improvement to take place. Also, the study examines how the questions in the course's assessment tests are linked to the course learning-outcome

objectives and Association of College and Research Libraries competency standards.

assessing, and improving all aspects of the course.

of distance learning in higher education.

**2. Literature review** 

Before examining assessment methods, the author briefly discusses background issues that the distance learning instructors of Texas Tech Library regularly address during the early stages of planning the assessments that are used in their course each year. These issues can be understood as questions that instructors must answer during the planning process. Some of the questions are what role will standards play in assessment, what learning theories underlie the assessment and instructional strategies, and how will assessment match the course's learning objectives and instructional strategies? The Texas Tech instructors' discussion of the issues and determination of the assessment methods to be used occur in the context of a structured yearly cycle of planning, developing, marketing, implementing,

Searches in several bibliographic resources using the keywords "higher education," "distance learning" or "distance education," and "assessment" yield a large number of articles, books, documents, and other materials on assessment. Some of these materials are guides, manuals, or action plans; articles on the need to integrate assessment into education; or reports of accreditation trends in higher education. Other materials found discuss strategies used to gain support for or to develop assessment programs, or report on statewide assessment programs of higher education curricula without the details of any particular assessment projects. The author found a less numerous group of articles that in some way or another discuss assessment of student learning in distance courses, including those that report the actual experiences of distance learning instructors in assessing student learning in their courses. Though a few of these articles report on studies that did not take place in a higher education environment, higher education faculty involved in distance learning should be aware of these studies because of their pertinence in the developing field

3 Web-based learning (also called online learning, e-learning, or electronic learning) comprises all forms of electronically supported learning and teaching. The information and communication systems,

whether networked or not, serve as specific media to implement the learning process.

One of these articles addresses the research needs of distance learning, and others describe recently developed online systems or approaches that assess in one way or another student learning outcomes. Oncu and Cakir (2011) examine the priorities and methodologies of research in online learning environments. The authors maintain that distance learning lacks research goals that, if observed, would lead to a better understanding of the impact of online learning environments on students. They propose four research goals for online learning concerned with learning achievement, engagement, and retention. Their goals are "(a) enhancing learner engagement and collaboration, (b) promoting effective facilitation, (c) developing assessment techniques, and (d) designing faculty development programs." The authors discuss some research work in these areas, and recommend research methods that are suitable for pursuing their goals. Su et al. (2011) report on the testing at three Taiwanese universities of an online portfolio assessment and diagnosis scheme (OPASS) that assists teachers in automatically assessing and diagnosing students' abilities in performing scientific inquiry. OPASS generates reports that diagnose learning problems and provide suggestions for improvement based on teacher-defined assessment input for the scientific inquiry experiment. Udo et al. (2011) propose a modified SERVQUAL instrument for assessing e-learning quality. The instrument measures five dimensions: assurance, empathy, responsiveness, reliability, and Website content. Analysis of the data collected from 203 elearning students who participated in a pilot study of the instrument revealed that four of the dimensions (excepting reliability) played a significant role in the students' perceived elearning quality. The authors point out that perceived quality affects student satisfaction with their e-learning, and is an indicator of future intentions to enroll in online courses. Sima et al. (2007) discuss the eMax Knowledge Assessment System developed by the Intelligent Knowledge Management Innovative Center of IBM Hungary and the John Von Neumann Faculty in Informatics at Budapest Tech. eMax can evaluate students' open-ended short answers including a few sentences or partially solved mathematical problems.4

A few online assessment systems or approaches are rather unique. Costagliola et al. (2009) discuss an approach to online testing that enables instructors to monitor learner behavior and test quality. The approach involves examining logging data related to learner interaction with the system during the execution of online tests and exploiting data visualization techniques to identify information useful for improving the assessment process. It focuses on discovering learners' behavior patterns and the conceptual relationships among test items. Hayes and Ringwood (2009) report on the development of a system used to authenticate telephone-based oral examinations. The authors assert that the system can, in turn, be used to confirm a student's ability in relation to submitted assignments and online test results and that it is an effective deterrent against plagiarism.

Portfolio creation is conducive to developing students' skills in documenting and tracking their learning, developing an integrated and coherent record of their learning experiences, and improving their self-understanding. Bhattacharya and Hartnett (2007) discuss a system that assesses the learning that takes place during the design and development of students' eportfolios. Caldarola and MacNeil (2009) review the similarities and differences in student cheating among the various assessment methods. They then investigate cheating indicators and predictors, and the methods available for detecting cheating on examinations. Also,

<sup>4</sup> Open-ended questions allow for a spontaneous, unstructured answer.

Assessment Methods of Student Learning

revise and improve the course.

standards is also demonstrated.

assessing, and revising an instructional program.

in Web-Based Distance Courses: A Case Study 189

and two or three sections in the spring. Each semester at least one of the course's sections is taught online to distance students. There are no classroom meetings for the distance students. The entire course is taught on the Internet using the Blackboard Course

Every term, the distance section of the course has been evaluated by students enrolled in the section in terms of the course content and instructor. The evaluations are done on printed, machine-readable forms. All credit bearing courses taught under the auspices of Texas Tech University must be evaluated using this form. The data collected on the forms is subjective and, in any case, is not the kind that can be used to assess what students have learned. Nevertheless, the course's instructors regularly consult student input recorded on the evaluations. This consulting takes place every year during the summer when the instructors

The instructors of LIBR 1100 have used more objective assessment methods to find out what their students are learning. They began measuring student learning outcomes with pre- and post-assessment tests in the fall of 2008 and have used the tests ever since. The intent of the tests is to determine as objectively as possible whether students enrolled in the distance section are learning what the instructors teaching the section intend for them to learn. Other assessment methods used in the online section of the course are practicum assignments that require the performance of skills, an annotated bibliography project, and quizzes. While the quizzes and pre- and post-assessment tests evaluate students' knowledge of the course's subject content, the practicums and annotated bibliography test the performance of skills.

This study briefly examines assessment methods that are suitable for Web-based courses, including online quizzes, tests, and exams; online discussions and other similar kinds of networked learning; problem-based learning and case-based reasoning; role playing; assessing practical skills; and laboratory experiences. The author then goes on to carefully examine those methods used to assess student learning in the Web-based distance learning course taught at Texas Tech University Library, highlighting experience gained by the instructors while developing and implementing pre- and post-assessment tests, online quizzes and exams, and authentic performance assessments that have been administered in the course. Student assessment data collected by all of the instruments employed in the fall of 2010 are analyzed, and this analysis is meant to demonstrate how the instructors' yearly cycle of developing and improving the course operates. The importance of linking assessment questions to course learning objectives and nationally recognized competency

The study's discussion of the various assessment methods used in the Library's online course represents a contribution to the limited documentation available in the professional literature on student learning-outcomes assessment projects that have taken place in the online distance learning environment. The assessments' findings, though relevant to one academic institution, nevertheless will enhance distance learning faculty's understanding of how the assessment process can be used to improve learning, teaching, course content, and delivery. Other important contributions are identification of assessment methods that work well in the online learning environment and an explanation of how assessment planning can fit into a yearly cycle that includes planning, developing, marketing, implementing,

Management System to students located across Texas and in some cases out-of-state.

they offer an analysis of the testing guidelines of the International Testing Commission and of several major universities, and a review and evaluation of several remote examination proctor systems in the United States along with recommendations for their use in various distance learning environments.

A limited number of actual assessments of what students are learning in online courses have been reported in the literature. Hufford and Paschel (2010) report on the experience gained by instructors at Texas Tech University Libraries while developing and implementing preand post-assessment tests that were administered in the distance learning section of a library research course taught in the fall of 2009. The assessment's findings were used to improve student learning, the course content, and teaching in the course's online section. Foster and Drew (2009) discuss student learning-outcome assessment that took place in an astrobiology course taught at the University of Florida. Pre- and post-tests along with "knowledge" assessments were used to evaluate the students' perceived and actual learning experiences. The course used both traditional classroom and distance learning technologies. The assessment's pre-testing indicated that the students had little prior knowledge of key astrobiology concepts. However, post-testing showed significant improvements in their comprehension of the subject. The assessment also revealed that, because of taking the course, the students developed confidence in writing on science topics as well as reading and understanding the primary literature of astrobiology. Brown and Hanks (2008) report on an innovative online workshop designed to assess and improve the writing skills of students enrolled in distance learning gerontology classes. Student learning assessment included Web-based writing evaluations in both pre- and post-tests. Urtel (2008) explores whether differences in academic achievement indicators exist between students taking a traditional classroom course and a distance learning course. The final grades; rates of D, F, and W grades; and end of term course and instructor evaluations of three-hundred and eight-five students enrolled in a course offered both in a classroom (116 students) and at a distance (269 students) were studied. The course content, instructor, textbook, and assessment methods were similar for both delivery formats. Student demographic information was included in the data analyses. Ivanitskaya et al. (2008) review the results of tests that assessed the library research skills of off-campus students. The investigators used a "Research Readiness Self-Assessment" test as a pre- and post-test in an off-campus Master's degree class at Central Michigan University. In particular, they investigated the impact that pre-tests have on the effectiveness of library instruction when students are given feedback on their pre-test performance. Similarly, Mulherrin et al. (2004) review the results of pre- and post-tests taken by students. The tests were taken by distance students enrolled in LIBS 150, a one-hour credit, elective library skills course offered at the University of Maryland. The tests were administered as one phase in the development of the course and proved to be an important factor in its eventual success.

#### **3. Aim and scope**

For several years now, Texas Tech University Library has offered a one-hour credit course titled "Introduction to Library Research" (LIBR 1100) to undergraduates. The course teaches the basics of library research and targets freshmen, though sophomores, juniors, seniors, and even an occasional graduate student enroll in the course. Most of the Information Services librarians participate as instructors. Several sections are offered each fall semester,

they offer an analysis of the testing guidelines of the International Testing Commission and of several major universities, and a review and evaluation of several remote examination proctor systems in the United States along with recommendations for their use in various

A limited number of actual assessments of what students are learning in online courses have been reported in the literature. Hufford and Paschel (2010) report on the experience gained by instructors at Texas Tech University Libraries while developing and implementing preand post-assessment tests that were administered in the distance learning section of a library research course taught in the fall of 2009. The assessment's findings were used to improve student learning, the course content, and teaching in the course's online section. Foster and Drew (2009) discuss student learning-outcome assessment that took place in an astrobiology course taught at the University of Florida. Pre- and post-tests along with "knowledge" assessments were used to evaluate the students' perceived and actual learning experiences. The course used both traditional classroom and distance learning technologies. The assessment's pre-testing indicated that the students had little prior knowledge of key astrobiology concepts. However, post-testing showed significant improvements in their comprehension of the subject. The assessment also revealed that, because of taking the course, the students developed confidence in writing on science topics as well as reading and understanding the primary literature of astrobiology. Brown and Hanks (2008) report on an innovative online workshop designed to assess and improve the writing skills of students enrolled in distance learning gerontology classes. Student learning assessment included Web-based writing evaluations in both pre- and post-tests. Urtel (2008) explores whether differences in academic achievement indicators exist between students taking a traditional classroom course and a distance learning course. The final grades; rates of D, F, and W grades; and end of term course and instructor evaluations of three-hundred and eight-five students enrolled in a course offered both in a classroom (116 students) and at a distance (269 students) were studied. The course content, instructor, textbook, and assessment methods were similar for both delivery formats. Student demographic information was included in the data analyses. Ivanitskaya et al. (2008) review the results of tests that assessed the library research skills of off-campus students. The investigators used a "Research Readiness Self-Assessment" test as a pre- and post-test in an off-campus Master's degree class at Central Michigan University. In particular, they investigated the impact that pre-tests have on the effectiveness of library instruction when students are given feedback on their pre-test performance. Similarly, Mulherrin et al. (2004) review the results of pre- and post-tests taken by students. The tests were taken by distance students enrolled in LIBS 150, a one-hour credit, elective library skills course offered at the University of Maryland. The tests were administered as one phase in the development of the course and

distance learning environments.

proved to be an important factor in its eventual success.

For several years now, Texas Tech University Library has offered a one-hour credit course titled "Introduction to Library Research" (LIBR 1100) to undergraduates. The course teaches the basics of library research and targets freshmen, though sophomores, juniors, seniors, and even an occasional graduate student enroll in the course. Most of the Information Services librarians participate as instructors. Several sections are offered each fall semester,

**3. Aim and scope** 

and two or three sections in the spring. Each semester at least one of the course's sections is taught online to distance students. There are no classroom meetings for the distance students. The entire course is taught on the Internet using the Blackboard Course Management System to students located across Texas and in some cases out-of-state.

Every term, the distance section of the course has been evaluated by students enrolled in the section in terms of the course content and instructor. The evaluations are done on printed, machine-readable forms. All credit bearing courses taught under the auspices of Texas Tech University must be evaluated using this form. The data collected on the forms is subjective and, in any case, is not the kind that can be used to assess what students have learned. Nevertheless, the course's instructors regularly consult student input recorded on the evaluations. This consulting takes place every year during the summer when the instructors revise and improve the course.

The instructors of LIBR 1100 have used more objective assessment methods to find out what their students are learning. They began measuring student learning outcomes with pre- and post-assessment tests in the fall of 2008 and have used the tests ever since. The intent of the tests is to determine as objectively as possible whether students enrolled in the distance section are learning what the instructors teaching the section intend for them to learn. Other assessment methods used in the online section of the course are practicum assignments that require the performance of skills, an annotated bibliography project, and quizzes. While the quizzes and pre- and post-assessment tests evaluate students' knowledge of the course's subject content, the practicums and annotated bibliography test the performance of skills.

This study briefly examines assessment methods that are suitable for Web-based courses, including online quizzes, tests, and exams; online discussions and other similar kinds of networked learning; problem-based learning and case-based reasoning; role playing; assessing practical skills; and laboratory experiences. The author then goes on to carefully examine those methods used to assess student learning in the Web-based distance learning course taught at Texas Tech University Library, highlighting experience gained by the instructors while developing and implementing pre- and post-assessment tests, online quizzes and exams, and authentic performance assessments that have been administered in the course. Student assessment data collected by all of the instruments employed in the fall of 2010 are analyzed, and this analysis is meant to demonstrate how the instructors' yearly cycle of developing and improving the course operates. The importance of linking assessment questions to course learning objectives and nationally recognized competency standards is also demonstrated.

The study's discussion of the various assessment methods used in the Library's online course represents a contribution to the limited documentation available in the professional literature on student learning-outcomes assessment projects that have taken place in the online distance learning environment. The assessments' findings, though relevant to one academic institution, nevertheless will enhance distance learning faculty's understanding of how the assessment process can be used to improve learning, teaching, course content, and delivery. Other important contributions are identification of assessment methods that work well in the online learning environment and an explanation of how assessment planning can fit into a yearly cycle that includes planning, developing, marketing, implementing, assessing, and revising an instructional program.

Assessment Methods of Student Learning

**4.2 Kinds of online assessment** 

**4.2.2 Online chats and discussions** 

**4.2.1 Quizzes and tests** 

planning.

in Web-Based Distance Courses: A Case Study 191

failure if they are ignored. A self-paced course may not be successful if collaborative projects or weekly class discussions are required. Independent project assignments may cause diminished participation in class discussions. Assigning weekly projects may cause delays and a backlog of work for the instructors. This last conflict could result in some students proceeding to subsequent assignments before receiving important instructor feedback and grades. In looking for conflicts and inconsistencies during assessment planning, plagiarism must be an ever present concern instructors of online courses must face. Are the assessment methods designed to discourage, if not eliminate, cheating. LIBR 1100 instructors have found the list of questions to be a very helpful tool that facilitates successful assessment

Additional considerations in planning assessment for online distance learning courses include the role communications technology will play. Does the technology facilitate the successful implementation of assessments? Multimedia technology provides multiple formats that appeal to a variety of learning styles. Graphics, images, maps, audio, and video used to convey course content or incorporated into learning activities can also be posted in online tests. Do these multimedia formats slow down the response rates of the system? Students want technology that works well and quickly, and enhances their learning. These

Instructors often assign students quizzes or tests to assess their knowledge of facts or understanding of concepts. They may or may not be timed, proctored, or graded. They can be taken online using a course management system like Blackboard, mailed to the students' homes, or proctored at some designated location by an authorized individual. Identity and security issues are major concerns that should be taken into consideration when planning a test that will be taken at a distance. Some instructors use security checks such as retinal scans; facial identification using thermographs; voice, palm, or fingerprint recognition; or, in the case of tests that require writing, analyses of the writing performed at intervals throughout the term. However, the technologies supporting these options can be expensive. Accessing quizzes and tests with passwords or generating a different test for each student who logs in may provide at least some degree of security in Web-based testing. However, these options are not completely secure against cheating. In the final analysis, the instructor never knows for sure who is taking tests at a distance. This drawback has also been an ongoing concern of instructors who teach in traditional classrooms. Nevertheless quizzes

Online chats and conference calls between students and instructor provide opportunities for discussion, negotiation of meaning, validation of understanding, and assessment. Some instructors place significant weight on the quality of student participation in these chat and conference discussions. They enable the instructor to discover what students are thinking and to determine if any of the students are having problems learning. With this kind of

information the instructor is able to give help on the spot and provide remediation.

technology concerns must also be addressed during assessment planning.

and tests constitute an effective way to test knowledge and understanding.
