BestChoice: Learning how to teach interactively over the web
Sheila Woodgate and David Titheridge
The University of Auckland, Auckland, New Zealand

The BestChoice web site (bestchoice.net.nz) was born in 2002 out of a desire to offer additional learning support to students in large first-year university courses. The aim was to create web-based activities that modeled a one-on-one interchange with an experienced teacher. Thus it was intended that content be developed systematically, using both information and question pages, and that users receive instructive feedback in response to their answers.

The activities currently on the web site have 24 000 opportunities for users to interact with the system and receive feedback. A selection (75) of activities is available in a menu accessed by clicking on DEMO MODE at bestchoice.net.nz. The design of these activities, the tools used to create them and the system that delivers them has been driven by the pedagogical requirements of teaching model, direct feedback from our users, and analysis of their usage data. The purpose of this communication is both to introduce you to the nature of the BestChoice activities and to share with you insights that have been gained as a consequence of our analyses of their usage by various cohorts of students.

THE IMPORTANCE OF HAVING A VERSATILE SET OF AUTHORING TOOLS
Our previous experience with developing computer-based learning activities led us to envisage a text-based system where images were used only when the pedagogy demands. We chose to develop our own authoring tools so that these could be modified as required by the teaching model. In order to achieve the goal that we had set ourselves, the author needed the capability to
The necessity for all of this (and more) is illustrated by a BestChoice question page for a drill-and-practice exercise where the user constructs and balances the half equations and the overall equation for the reaction between iron(II) ions and dichromate ions. The page loads as shown below.



The objective was to create an on-screen version of the pen-and-paper approach (balance atoms other than O and H, balance O, balance H......). This requires 23 answers. 10 are formulae for the species involved, 13 are numbers.

Students learning to balance redox equations on pen-and-paper would have all chemical formulae available so that they can focus on the balancing. Thus the formulae are on dropdown lists in the corresponding web-based exercise.

RedoxEquation 2

To simulate the balancing sequence, the completion of each part (for example, the balancing of chromium) triggers appearance of the next part (for example, the list from which water would be chosen).

On paper, once the correct formula for a species is written, the stoichiometric coefficient is entered. The screen shot shows that in the web-based balancing, choosing a formula from the dropdown triggers appearance of a text box into which a number can be entered.



Once the half-equations are complete, they are added to give the overall equation. In the overall equation, the order in which the reactants (and products) are chosen from the dropdowns should not matter. In the web-based balancing, this requires that each coefficient-answer is coupled to its formula-answer.

RedoxEquationFinished

Finally for the completed page to have a text-box appearance, all buttons used to access the dropdown lists, all text-surround boxes and all coefficients of 1 must have disappeared on display of the correct response.

While the page above simulates fairly well the balancing of a redox equation with pen and paper, a significant difference between this exercise and the pen-and-paper one is that in the web-based exercise, incorrect answers are exposed immediately and can be corrected.  Thus the user has an opportunity both to learn from his/her mistakes and to proceed to complete a correct balanced equation.

While every page authored informs the authoring of future pages, our priority has been to let the content drive the style of question page created. Some page styles (like the redox equation one) are re-used. However, each system has its unique features and pedagogical possibilities, and the point of BestChoice is to go beyond drill and practice to expose the thinking behind the problem, placing a greater emphasis on the problem-solving process than on the overall answer.

THE IMPORTANCE OF COLLECTING DATA AND ANALYSING THAT DATA
The users of BestChoice are in equal parts New Zealand high school students, UK high school students and New Zealand first-year university students. 80 000 users have entered 30 000 000 correct answers on BestChoice pages. We have learned that students can be a great help in designing systems to teach them. With the best will in the world, we cannot put ourselves in their shoes. We can, based on our experience as teachers, create activities that we think that students will like and deliver these in what we consider to be a learner-friendly way, but students should be the judge and jury. They are the customers.

With a view to encouraging users to give us feedback, a survey was placed on the last page of every Bestchoice activity. Users may enter a Likert scale rating (1 to 6) and/or a text-comment. This is one of the best things that we ever did.  24 000 comments relevant to teaching and learning have been entered along with 122 000 Likert scale ratings. Overall, 79% of the comments are compliments or suggestions, 21% are criticisms. 30% of the Likert scale module ratings are the maximum rating, and 79% of responses are positives.

Completion of the survey is voluntary, so it is important also to collect usage data for all users. A companion application (BestChoice Reports) was developed to view these data. This application is both used by the developers and also made available to teachers of students who are BestChoice users.

User comment has informed the approach taken when constructing the activities.
The first activities developed used an enquiry approach. The user was given minimal information on screen pages preceding the question pages, and content was developed through the questions and feedback. Comments from users indicated that they liked the integration of information pages, questions and feedback.

13 Mar 06 I really like this because you have a review and the questions are right after it and everyone knows you need to do alot of questions to know the material. Keep it up!

31 Mar 05 This is really an awesome site, it has helped so much especially with the step by step instructions and the helpful explanations after you get a question right. The exclamation mark when you ask for help is good too cos its like ah hello, its this obvious. I love this site.I can see it is going to help me immensly this year

However, users did not like being asked about something had not featured in earlier information pages.

13 Mar 06 i really liked how there were notes to read throughout the quizes. Maybe for future quizzes notes could be added before the question is asked to give hints as to what the answer will be

This feedback led us to alter our module format to supporting users with information pages prior to the relevant question pages.  This type of activity has a better completion percentage and receives more favourable student comment. Students like to be able to flick back to information pages in BestChoice without losing their previously-entered answers.

In a similar vein, carefully-worded instructions are important so that users know what to expect when they embark on entering/choosing answers on a question page. Even with experience, the author does not always anticipate the detail of instruction needed as per the comment below that pertains to a recently-authored page.  The student was emailed, and it transpired that he had a perfectly valid point, and a further instruction was added.

3 Oct 10 instructions on what is required to be done is highly unclear. This causes some answers to be marked wrong despite the answers technically being correct, only because the answers have been entered incorrectly. I repeat, the INSTRUCTIONS on what the question requires, particularly on HOW the question needs to be answered, are usually unclear and/or incomplete. Highly frustrating...

User comment has had considerable impact on our marking practices
Scores are important to students, even though completion of BestChoice activities commonly does not contribute to course assessment. Our first scheme was one mark for each BestChoice answer irrespective of whether it was entered correctly on the first attempt or on a subsequent one. Each answer was marked instantly and feedback was displayed adjacent to the answer. The feedback feature was (and is) very popular. The instant marking was much less popular for reasons pointed out in the two comments below.  As a consequence, instant marking is no longer our default marking style.

9 Oct 10 for the last question i think it would be best if the answer didnt auto correct as with the previous ones. Sometimes you dont understand the reaction scheme until youve played with combinations for a while.

9 Oct 10 i prefer it when i can manually press 'mark answer' so i can correct myself when i accidentally click the wrong answer

Additionally, users told us that they did not think that it was fair that they got the same mark for a first-right answer as they did for a changed-right-answer. Thus, in 2007 we put in place the system that currently operates. The overall score is still for completion, but this is now the sum of first-right-answers and changed-right-answers, with the marking bar displaying both of these.

This development, driven by student feedback, has been a huge bonus for our evaluation program because the percentage of first-right answers, in conjunction with the number of Give-Ups, gives us evidence for (a) whether a question is achieving its objective for most users and (b) whether the feedback for a wrong answer is sufficiently instructive to enable most students to recover from their initial mistake.

Survey and usage data guide us in providing scaffolding appropriate for a particular topic.
Our intention was to create activities to help students bridge the gap between being presented with information (listening to a lecture or reading a book) and flying solo where they solve multi-step problems or apply concepts without support. Thus BestChoice activities ask students lots of little questions that would not normally feature in written exercises.  The aim  is to probe whether the student understands the background to the system in the problem.  This approach means that there are lots of opportunities to give users feedback at a point when they are receptive due to just having engaged with the content.  The usage data that accrues give us clues where understanding breaks down.

It is important that the questions asked include content that authors perceive to be the obvious because what is obvious to us is not necessarily obvious to learners.  These questions also build student confidence, showing them that they do have some of the prior knowledge required to be successful.  However, surprises abound when one takes this approach and then examines the data. 

It is pleasing, for example to know that users do not find balancing equations difficult, provided they are given the reactants and products. On the other hand, it was disturbing to find that 30-40% of students do not know that 2Cl(g) is a higher energy system than Cl2(g).  Likewise, who would have believed that students who had happily identified tetrahedral, trigonal planar and bent shapes from static pictures of mononuclear species would not cope nearly so well with identifying the same shapes at particular atoms in a multi-nuclear Jmol of an amino acid?

It is important to go beyond simply looking at the data, to using it to inform modifications that may make the question more accessible. This could involve building in repetition where knowledge of concepts is probed from various perspectives. For example, users can be asked to identify the nucleophile and identify the electrophile many times before they start to get the idea.

Despite our general emphasis on supporting students, there are areas where, based on usage data, we have removed scaffolding. Stoichiometry is one of these areas. In our early stoichiometry question pages, users worked through problems in a linear way, choosing/constructing the relationships required, entering the given data into the relationships on-screen and then calculating and entering the answers.

Our first attempt to go beyond what was a somewhat over-scaffolded approach was to introduce problem-planning pages before the number-work pages.  This was consistent with our philosophy of fostering development of transferrable skills. Users were asked to identify the unknown and known as well as the quantities that needed to be calculated and the relationships required to calculate these.  Usage data revealed that students found the planning pages much harder than the numbers pages.

The introduction of planning pages had obviously significantly increased the number of answers that needed to be entered to solve the problem, so the next step was to reduce the scaffolding on the number-entry section, making the assumption that if students had in front of them a simple mathematical relationship (x = yz or x = y/z), and values for y and z, they could calculate the answer. This achieved a reduction in the number of answers to be entered and enabled integration of number-entry and planning sections onto the same page.

We have also moved away from revealing answer fields one at a time (as in the redox equation page), to revealing answers in blocks: first the known and unknown, second all of the intermediate quantities, third all of the relationships, fourth all of the calculation fields. This gives users the possibility of working forward from the known or backward from the unknown. The screen shots show, for one page, various stages in the answering process.

On loading:


Planning complete:


Calculation complete:
Note the annotations that are displayed on screen as feedback for correct answers.



We have data from a variety of cohorts for a three-problem sequence, the one above and two others based on reactions with different stoichiometries. The screen shot shows, for a class of first year university students working through the sequence, that % first-right answers and % completion improve, and time taken diminishes. 



While the data above are encouraging, the general area of stoichiometry continues to challenge us as both % completions and survey feedback on this type of activity are somewhat below BestChoice norms.  We continue to experiment with different approaches.


In conclusion
The possibilities that the web offers for experiments in instructional design, like BestChoice, are beyond what can be imagined.  There is so much potential to complement, extend and inform conventional modes of instruction. Our voyage of discovery in the area of interactive learning has given us a different perspective on teaching and learning, and enabled us to connect with our students in ways that we would not have previously thought possible. The result is a reciprocal learning situation where in reward for our efforts to support their learning, students support us by providing good suggestions and encouraging comments, such as those below. 
12 Oct 10 Extremely helpful in discovering my weaknesses in this topic and also at helping me overcome them.
6 Oct 10 That was really good! I had no idea on this topic, and by slowly going through it, I was learning then practising and now I mastered it! Thanks!

We would encourage to have a look at BestChoice (bestchoice.net.nz)In addition to the 75 learning activities mentioned above, there are two presentation modules in the DEMO mode menu, under the heading BCCE.  These show in more detail how we use our data.  The blue text on the pages are links to more information or data (sometimes at the bottom of pages). If you choose to SIGN UP in order to have a more valid user experience, Choose Other as your Institution and check out the General courses. 

Lastly, if you are involved in any web-based education initiatives, do set up automatic data collection.  It is like opening your eyes after years of working blind.

We would like to acknowledge financial support from The University of Auckland, the New Zealand Ministry of Education (2006-9) and the Royal Society of Chemistry (2006-10).


Return to Fall 2010 CCCE Newsletter