DESIGNING QUESTION PAPER MARKS DISTRIBUTION BASED ON BLOOM’S TAXONOMY LEVEL FOR COURSE OUTCOMES MEASUREMENT

Swarna M Patra 1*, Ramakanth Kumar P2, K N Subramanya3

1Department of Chemistry, RV College of Engineering, Bengaluru, India

2Department of Computer Science and Engineering, RV College of Engineering, Bengaluru, India

3Department of Industrial Engineering & Management, RV College of Engineering, Bengaluru,India

*Corresponding Author’s Email: swarnamp@rvce.edu.in

ABSTRACT

Outcome based education is an innovative teaching and learning methodology continuously focusing on the improvement of learning outcomes. Knowledge, skill, and competence learning Outcomes are required to achieve during the process of learning. Outcome based assessment is the assessment process which is aligned with the learning outcomes. This paper represents an insight to formulate the question papers as per Bloom’s Taxonomy level. An application of the theoretical framework for the integration of question paper setting through achievable matrix is described. Uniformity and systematic methodology are implementation is one of the key requirements of the accreditation agencies. Customization of the achievable matrix and implementation for different courses will bring uniformity across the institutions.

Keywords: Outcome Based Education; Learning Outcomes; Achievable Matrix; Course Outcomes

INTRODUCTION

In 21st century the need of the hour is enhancing the learning capabilities of the learners (Basri et al., 2004). Learning outcome indicates a learner’s knowledge, understanding and capable of doing after the completion of a course or program. Towers cited the Outcome Based Education (OBE) focused on outcomes that are measurable (Charles & Towers, 1996). It includes knowledge, skill, and competence that a learner should gain on successful completion of a module of learning. Outcome of the assimilation of information obtained during learning (E. Commission, & Culture, 2008) is known as knowledge. Skill can apply knowledge and use know to complete tasks or solve problems (E. Commission, & Culture, 2008). Use of knowledge and skills at work or study situations for professional/personal development (E. Commission, & Culture, 2008) is known as competence. These attributes must have been acquired by the learner on successful completion of the course (Jaafar et al., 2008; Rashid et al., 2008). In OBE measurement of learning outcomes is adopted from American Accreditation Board of Engineering and Technology (ABET) (Bellis et al., 2000). Various teaching methodologies are adopted to improve the deep learning of the students (Patra & Subramanya, 2018). Mostly summative examinations are conducted for year/semester end. Summative examination does not give scope for improvements for the students in their process learning (Subheesh & Sethy, 2018). On the other hand, Assessment For Learning (AFL) is a form of assessment which does not give grade or score but targets to identify the strengths, weaknesses and needs of student to improve learning. As a result, students are better prepared for summative assessment (Mokhtar & Wan Adnan, 2017). Yet, higher education institutions have not adopted unique standard method for the measurement of learning outcome. This is to provide freedom to educational innovation, regional factors and to express of the unique characteristics of the individual institutes (Canadian Engineering Accreditation Board, 2016). Systematic assessment procedure is mandatory for all higher education accrediting agencies (Cartwright, Weiner, & Streamer-Veneruso, 2010). The success of OBE depends on alignment of learning activities with the intended learning outcomes and the assessment tasks alignment with the intended learning outcomes. In an educational institution transparent assessment process is required to quantify the student's learning and performance.

A student’s assessment in study for qualification and required competency is ensured by the transparency of assessment process. The evaluation process must be aligned with the requirements of the outcomes, teaching, and learning in outcome-based education. Effective learning needs effective teaching strategies. The five effective strategies of teaching are described in (Wiliam & Thompson, 2007). The five key strategies of teaching are i. to Clarify, to share and to understand success and goals for learning of learners ii. Student’s learning can be monitored by effective classroom discussions, questions, and team activities iii. To provide feedback or input for forward learning, iv. Making students responsible their own learning, and v. Helping students to learn from one another.

In Outcome Based Education learning outcome means a learner is expected to know, understand and able to do by successful completion of the course or program. Programme Learning Outcomes or Programme Outcome (PO) refers to the attribute students can do or exhibit once their course of studies is completed successfully (EAC, 2007). Thus, learning happens at different stages. Program learning outcomes (EAC, 2007) indicate areas of learning expected at the end of a program. As per outcome-based education, the Accreditation Board has articulated the 12 Graduate Attributes (the program learning outcomes). To measure learning effectively the alignment of program outcomes, course outcomes with Bloom’s Taxonomy (Bloom et al., 1956) is very much required. Students achieving the intended learning outcomes are investigated through measurement of course outcome attainment (Patra & Ramakanth, 2016). The framework and model of this paper is to target on the uniform, systematic distribution of marks in the question paper for different levels of learning.

RESEARCH METHODOLOGY

One of the basic questions facing by educators of all time is “Where do we begin in seeking to improve human thinking?” (Houghton, 2004). Bloom’s taxonomy is used by the educators as guidance towards the assessment development or formulation (tests and evaluations of student learning), curriculum (lessons, projects, and other activities related to learning), and instructional methods for questioning strategies. Bloom developed a classification method for thinking behaviors that are believed to be important in the processes of learning. This framework is referred as three domains of Bloom’s taxonomy corresponding to cognitive, affective, and psychomotor.

In the last two or three decades all over the world Outcome Based Education has gained popularity in Engineering education. There has been a paradigm shift to emphasize the learning outcome of the students. A strong initiative is taken by Government of India in introducing outcome-based education for engineering education to enhance the quality of national workforce. A novel skeleton and reference method is designed focusing on psychomotor skills-based domain, consisting of six levels. The method is implemented in the outcome-based education environment for course outcomes assessment. The six levels learning of revised Bloom’s Taxonomy is given in Figure 1.

Figure 1: Revised Bloom’s Taxonomy

One of the key features of this method is distribution of marks with respect to course outcomes in question paper is done according to Bloom’s Taxonomy level. The course outcomes for Engineering Chemistry are designed focusing on Bloom’s Taxonomy level. The course outcomes for Engineering Chemistry 18CH12/22 are given below:

Bloom’s Taxonomy is a two-level model for classifying thinking accounting to six hierarchical complex cognitive levels. The lower order skill consists of knowledge, understanding, and application. The higher order skill consists of analysis, create, and design. The weightage for CO1 and CO4 is given less compared, compared to CO2 and CO3. Questions related to CO4 are mostly dealing with the societal problems. These contents are not taught directly in the curriculum. This puts a challenge on the learner’s depth of understanding. The CO3 questions are focused mostly on the problems or analysis of the theory or concept taught. Basic understanding and pre knowledge are captured in CO1 and CO2 questions.

Marks Distribution Details

The block diagram, shown in the Figure 2, depicts the relationships between different OBE terminologies.

Figure 2: Relationship of OBE Terminologies

Distribution of marks to all the COs units wise in course

In Engineering Chemistry (18CH12/22) the total theory marks is 100. Theory marks further split into assignment (20 marks) and syllabus (80 marks). So, the 80 marks is distributed among 5 units and converted to percentage as shown in table 1.

Table 1: Distribution of Marks CO Wise

Conversion of CO percentage to quiz and test marks

Distribution of percentage of marks to Quiz and Test. Conversion of percentage of CO marks to actual CO marks for question paper. Table 2 shows the splitting of CO percentage and converting to marks.

While setting up the question paper the CO marks are rounded up to match the total number.

Table 2: Distribution of CO Percentage and Conversion of Percentage to Marks

Implementation

The implementation of Achievable Matrix is done for Engineering Chemistry (16CH12/22 and 18CH12/22). We prepare the Achievable Matrix at the beginning of each semester. We get a clear picture as how to set the question paper as per the course outcomes. The question paper is prepared and thoroughly scrutinized by the faculty members.

The quiz and test question papers are prepared using Achievable Matrix. Test is conducted and the answer booklets are evaluated. After evaluation, the marks are entered in a excel sheet in two parts for CO attainment. The first part is the question to CO mapping matrix given in table 3 for Quiz. The matrix has elements 0 and 1 entries. If the question does not correspond to CO, then the matrix element is zero. If question corresponds to one CO completely then matrix element is 1 or if it belongs to more than one CO then weightage will be decided in such a way that, the total weightage should be 1. In the second part the Quiz marks obtained by individual students are entered in table 4. Using table 3 and the individual student’s marks the CO attainment is calculated. The CO attainment for CO1 to CO4 is represented as CO1A, CO2A, CO3A and CO4A. The percentage of CO attainment is calculated by dividing the total marks obtained in a particular CO to the maximum CO marks of the corresponding CO in the question paper and multiplying with 100. The CO attainment for the entire three Quiz is done separately. Like Quiz the Test Question – CO mapping matrix is setup and individual students mark obtained is entered to calculate the CO attainment for Test. The CO attainment is calculated for all the three tests. The arithmetic average of Quiz and Test CO attainment is considered as the theory CO attainment of the course. Like theory CO marks distribution, we have done the CO marks distribution through Achievable Matrix for experiments and assignments. After calculating the CO attainment of individual students for theory, lab, and assignment the average is taken over all the students enrolled in that course.

Table 3: Quiz Question – CO Mapping Matrix

Table 4: Quiz Marks and CO Attainment

Further the CO attainments are segregated according to branch, semester or academic year depending on the need. Action plan is prepared based on the course outcomes attainment to enhance the learning skill.

DISCUSSION

Assessment is critical in acknowledging learner has gained the knowledge, skills and competences as intended while designing the course. Course Outcome marks distribution described here has successfully implemented for Engineering Chemistry course (Canadian Engineering Accreditation Board, 2016; EAC, 2007). This can be customized for other courses easily. It is difficult to monitor the learning outcomes progress of learner without the uniform CO distribution marks as per Bloom’s Taxonomy in question paper (Bloom et al., 1956). After the test is conducted course attainment is calculated. We analyze the attainments and set the action plan. While setting the action plan we relook into the achievable Matrix to see if any changes are required. Initially it appeared as a challenge for the instructor to set question paper according to achievable matrix. It was overcome easily by preparing innovative questions to cover the COs. An excel sheet is prepared to calculate the CO marks distribution automatically. The automated excel sheet can be made available for customization on request.

CONCLUSION

In the internal quiz/test the pattern of question indicates the intention of asking it. Questions are broadly classified as ‘open’ or ‘closed’. Closed questions have one clear answer and are useful to verify recall, understanding during explanations and in recap sessions. To help students to develop higher order thinking skills, open questions are required, which allow the students to give a variety of possible responses. A question paper should have balanced weightage of open and closed and questions for all the levels of students. Achievable matrix set up is very much essential to address the marks distribution ratio in the question paper as per Bloom’s Taxonomy Level. While focusing on the mark allocation to covers all the course outcomes which is designed for the course, the proposed method will be able to help in continuous quality improvement of educational institutes. The developed method ensures to capture the essential characteristics of learning outcome. This led students to answer questions which demand lower order thinking skill or less sophisticated thinking skill along higher-order critical thinking skills. Independently the method can be used for any program/course or target group. Authors strongly believe in future this method will encourage and provided the detailed taxonomy assessments with illustrative examples of scenarios for the practical use of the model. Implementation of Achievable Matrix to set up question papers at program level will bring uniformity across the institution.

Conflict of Interests

The authors declare that they have no conflict of interest.

Acknowledgement

The authors are thankful to the institutional authority for completion of the work.

REFERENCES

Basri, H., Che Man, A.B., Wan Badaruzzaman, W.H., & Nor, M.J.M. (2004). Malaysia and the Washington Accord: What it Takes for Full Membership. International Journal of Engineering and Technology, 1(1), 64-73.

Bellis, I. (2000). Delivering HRD in the new millennium-are we taking it there or being taken? SANDF Bulletin for Educational Technology 23, pp 13-24.

Bloom, B.S., Engelhart, M.D., Furst, E.J., Hill, W.H. & Krathwohl, D.R. (1956). Taxonomy of Educational Objectives. David McKay Company, Inc., U.S.

Canadian Engineering Accreditation Board. (2016). Accreditation Criteria and Procedures. Engineers Canada. Retrieved From: https://engineerscanada.ca/sites/default/files/accreditation-criteria-procedures-2016-final.pdf

Cartwright, R., Weiner, K. & Streamer-Veneruso, S. (2010). Student Learning Outcomes Assessment Handbook, Montgomery College, Montgomery County, Maryland. Retrieved From: https://www.scribd.com/document/426034576/Learning-Outcomes-Assessment

Charles, T.G, & Towers, J.M. (1996). An Elementary School Principal's Experience with Implementing an Outcome-Based Curriculum. Contemporary Education, 68(1), pp 67-72.

E. Commission, & Culture. (2008). The European Qualifications Framework for Lifelong Learning (EQF). Office for Official Publications of the European Communities, Luxembourg.

EAC. (2007). Engineering Programme Accreditation Manual. Engineering Accreditation Council, BEM. Retrieved From: https://www.eac.org.my/web/document/EACManual2007.pdf

Houghton, W. (2004). Engineering Subject Centre Guide: Learning and Teaching Theory for Engineering Academics. HEA Engineering Subject Centre, Loughborough.

Jaafar M.S., Nordin, N.K., Wagiran, R., Aziz, A., Noor, M.J.M.M., Osman, M.R., Noorzaei, J. & Abdulaziz, F.N.A. (2008). Jaafar M.S., Nordin, N.K., Wagiran, R., Aziz, A., Noor, M.J.M.M., Osman, M.R., Noorzaei, J. & Abdulaziz, F.N.A. (2008). Assessment Strategy for an Outcome Based Education. International Conference on Engineering Education, July. Retrieved From: https://www.semanticscholar.org/paper/Assessment-Strategy-for-an-Outcome-Based-Education-Jaafar-Nordin/d041dd7a06d1692ec261bf32a2b7325f122b5863

Perception on Assessment for Learning in Outcome-Based Education Curriculum: A Case Study in the Optical Communication Course. 7th World Engineering Education Forum (WEEF), 13-16 Nov. Retrieved From: https://ieeexplore.ieee.org/document/8467020

Patra, S.M. & Ramakanth, K.P., (2016). Academic Excellence through attainment of course outcomes a case study on Engineering Chemistry. Journal of Educational Research and Medical Teacher, 4(1), pp 9-19.

Patra, S.M., & Subramanya, K.N. (2018). Blended learning is a Journey not an Event.International Journal on Recent Trends in Business and Tourism, 2(4), 19-23.

Rashid, R.A., Abdullah, R., Zaharim, A., Ghulman, H.A. & Masodi, M. (2008). Engineering Students Performance Evaluation of Generic Skills Measurements: ESPEGS Model. 5th WSEAS / IASME International Conference on Engineering Education (EE'08), July 22-24. Retrieved From: file:///C:/Users/kabir/Desktop/education605WSEAS.pdf

Subheesh, N.P. & Sethy, S.S. (2018). Assessment and Evaluation Practices in Engineering Education: A Global Perspective. 3rd International Conference of the Portuguese Society for Engineering Education (CISPEE), 27-29 June. Retrieved From: https://ieeexplore.ieee.org/document/8593451/authors#authors

Wiliam, D. & Thompson, M. (2007). Five 'Key Strategies' for Effective Formative Assessment. National Council of Teachers of Mathematics. Retrieved From: https://www.wagnerhigh.net/ourpages/auto/2015/2/2/52792377/Five%20Key%20Strategies% 20for%20Effective%20Formative%20Assessment.pdf