Evaluation

**What is evaluation?**
One of the three phases of the systematic development of online learning as proposed by Dabbagh & Bannan-Ritland (2005) is evaluation. According to Clark (2000) cited in Dabbagh & Bannan-Ritland (2005:234), '**[e]valuation** has been characterized as a process used to judge the "**worthwhileness**” of something in order to make decisions' (own emphasis). This is considered different to what assessment is.

Since I have developed a prototype of an online learning intervention and would like feedback that will help me improve it, I have decided to adopt an evaluation process. I have also decided to use //formative// instead of //summative// evaluation. Tessmer (1993) cited in Dabbagh & Bannan-Ritland (2005:257) says '[f]ormative evaluation is charaterized as "a judgment of the strengths and weaknesses of instruction in its developing stages for the purpose of revising the instruction to improve its effectiveness and appeal"'. Tessmer (1993) further highlights that prominent formative evaluation methods include (i)expert review, (ii)one-on-one evaluation, (iii)small group, and (iv)field test.

Dabbagh & Bannan-Ritland (2005:235) have characterized a systematic process of evaluation by the following four steps:


 * 1) Clearly determine the purpose, desired results, and methods of evaluation.
 * 2) Formatively evaluate the design and development prior to launching the online design.
 * 3) Revise the online materials according to the results of the formative evaluation.
 * 4) Implement the online learning experience and evaluate the results according to the identified goals.

Another model for formative evaluation is illustrated in by Morrison, et al's (2007:308) Figure 12-1. For the purpose of this intervention, I will not follow this model. Although, I will be adopting Dabbagh & Bannan-Ritland's systematic process of evaluation, it is however beyond the scope of this intervention to **revise** the online materials (Step 3) and **implement** the online learning experience (Step 4). The following paragraphs will illustrate Steps 1 and 2.

Step 1: Clearly determine the purpose, desired results, and methods of evaluation.
(a) The purpose of this evaluation is to determine from (b) The desired results are (c) The method of evaluation
 * the mathematics education specialist whether the online learning intervention is warranted
 * the online designer whether the online material might need to be improved
 * the mathematics education specialist to suggest appropriate activities for a particular course
 * <span style="font-family: Georgia,serif; font-size: 16px;">the online designer to indicate areas of improvement to the design
 * <span style="font-family: Georgia,serif; font-size: 16px;">Used Google docs to create a questionnaire that will help me ormatively evaluate the design and development prior to launching the online design.As suggested by Clark and Mayer(2003), cited in Dabbagh & Bannan-Ritland (2005:235), there may be formal and informal ways to evaluate online learning. In order to formatively evaluate the intervention, I have developed a [|questionnaire] to help me improve the prototype. This questionnaire was completed by both a mathematics teacher who is also an online designer who satisfies the learner profile, and a mathematics education specialist who satisfies the lecturer profile as described under Exploration.

Step 2:<span style="font-family: 'Cambria','serif'; font-size: 16px;">Formatively evaluate the design and development prior to launching the online design.
<span style="font-family: Georgia,serif; font-size: 120%;">The [|results] from the participants were then converted from Google Forms to //Microsoft Excel// as indicated below. <span style="font-family: Georgia,serif; font-size: 120%;">Series1=Education Specialist; Series2: Online Designer <span style="font-family: Georgia,serif; font-size: 120%;">When I received the results from the Education Specialist I was a bit dispondent, and I decided to call him immediately. He was ecstatic and he could not believe that there could be something like it. He told me he gave me 10 on everything, and said that it could be something he could use for his course next year. I put the phone down in disbelief because he found there is no need to improve on it. The questions running in my mind were, did he give me 10 <span style="font-family: Georgia,serif; font-size: 120%;">I was however not surprised with my colleague's results. He scored the prototype low on Appearance. He felt that I needed to improve on the use of font size and the of images. I do not agree with him though that I do not understand the learning outcomes, for example.
 * <span style="font-family: Georgia,serif; font-size: 120%;">because he liked it or
 * <span style="font-family: Georgia,serif; font-size: 120%;">because he would like me to pass?

<span style="font-family: Georgia,serif; font-size: 120%;">I had missed a lot of opportunities in developing this prototype only because I had to learn to use Moodle fast enough to make the deadline. As a result the only way I could get around was to at least integrate the context through the kind of questions I asked, such as learners knowing about representing integers by:
 * <span style="font-family: Georgia,serif; font-size: 120%;">My own evaluation: **
 * <span style="font-family: Georgia,serif; font-size: 120%;">withdrawing money and depositing
 * <span style="font-family: Georgia,serif; font-size: 120%;">temperature below and temperature above certain degrees

<span style="font-family: Georgia,serif; font-size: 120%;">This is one of the examples I had developed, but had to let go of it. Only because I did not know how, and time was against me:

<span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;"><Start>This table shows the recorded minimum and maximum temperatures for Johannesburg. This is real data.
 * || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Jan || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Feb || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Mar || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Apr || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">May || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Jun || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Jul || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Aug || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Sep || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Oct || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Nov || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Dec ||
 * <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Max (°C) || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">35 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">34 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">32 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">29 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">26 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">23 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">24 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">26 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">31 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">32 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">33 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">32 ||
 * <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">Min (°C) || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">7 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">6 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">2 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">1 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">-3 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">-8 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">-5 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">-5 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">-3 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">0 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">2 || <span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">4 ||

<span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">//Source: South African Weather Service//

<span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">a) What was the coldest month as recorded by the South African Weather Service?

<span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">b) What was the warmest recorded month?

<span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">c) Which month had the smallest //range// in recorded temperatures? (The range is the difference between the maximum and the minimum temperature).

<span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">d) Which month had the largest range in recorded temperature?

<span style="background-color: #1400ff; color: #ffffff; display: block; font-family: Georgia,serif; font-size: 16px; text-align: left;">e) Complete the white cells by entering the number and its sign to get the number on the right.

<span style="background-color: #1400ff; display: block; font-family: georgia,serif; font-size: 16px; text-align: left;"><End>
 * <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 21px; text-align: left;">7 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">-1 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">6 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">-4 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">2 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">a || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">1 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">b || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">-3 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">c || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">-8 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">d || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">-5 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">e || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">-5 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">f || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">-3 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">g || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">0 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">h || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">2 || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">i || <span style="background-color: #1400ff; display: block; font-family: 'cambria math','serif'; font-size: 16px; text-align: left;">4 ||

<span style="font-family: Georgia,serif; font-size: 16px;">The strength of the Activity for me would be to make this available for others to learn from it, and has made me realise what my next challenge is, to develop this further. The weakness is that it did not make explicit the kinds of things activities learners could engage in, because I had realised there was no longer going to be any online access to it. Thank you to Andre Hess, the web developer, who made it possible.

<span style="font-family: Georgia,serif; font-size: 16px;">I do not see how the learning outcomes could have been changed, they are set by the Department of Basic Education. I thought about having the questions randomised, but I wanted the questions to do the scaffolding, as I had no other medium, such as "ask the expert". So the questions are arranged in that way to develop the concepts. I started using a Glossary but when I realised the work was not going to be online, I stopped developing instruction for them. also would have like to use images more, as I believe people learn better that way.

<span style="font-family: Georgia,serif; font-size: 16px;">The unintended and unanticipated learning outcomes were:
 * <span style="font-family: Georgia,serif; font-size: 16px;">I did not realise this would strain my working relationship with one of my colleagues
 * <span style="font-family: Georgia,serif; font-size: 16px;">The good news though is that we are now back to where we were, talking politics. You may have noticed that he is one the respondents for pototype questionnaire. And he has applied to study the same course next year.
 * <span style="font-family: georgia,serif; font-size: 16px;">Moodle is now available in our Juta intranet for any staff member to use.

<span style="font-family: Georgia,serif;">References
<span style="font-family: Georgia,serif; font-size: 16px;">Dabbagh, N & Bannan-Ritland, B (2005) //Online Learning: Concepts, Strategies, and Application.// Upper Saddle River, New Jersey: Pearson Merrill Prentice Hall

<span style="font-family: Georgia,serif; font-size: 16px;">Morrison, G R, Steven, M R, Kemp, J R (2007) //Designing Effective Instruction// 5th Ed. New York: John Wiley & Sons, Inc.