19 June 02005

Evaluation of Learning Activity Management Systems

I've finally finished the rigorous evaluation report of Learning Activity Management Systems (LAMS). Seb Schmoller has an overview of the report and commentary on the small number of actual LAMS implementation cases.

One strand of the report jumped out at me. It observes that "it is less easy to adapt [a] lesson 'on the fly' in LAMS than in a traditional teacher-facilitated session," and that "some [students] were frustrated by the inflexibility of a LAMS sequence". Elsewhere the report refers to the linearity of LAMS sequences as restrictive and less than satisfactory.

What this suggests to me is that LAMS — in common, it must be said, with most e-learning approaches — reinforces the separation between the planning of a learning experience and its execution. This separation reduces the scope to be sensitive to the interactions with (and between), and to adjust and improvise accordingly.

In my last posting on LAMS I said that the software was making tutors into programmers of learning. The evaluation report is consistent with that, but also evokes some of the debates that took place in human-computer interaction ten to fifteen years ago. In particular, I'm thinking of Lucy Suchman's book Plans and Situated Actions, which studied people's interactions with an advanced photocopier (!) that used an expert system to guide its users. While the designers of the expert system worked on the basis that people started out with a fully-realised plan of what they wanted to do and then implemented it, Suchman's research showed that the users tended to have detailed plans only one step ahead and fuzzy ones beyond that, which allowed them to improvise and adapt as their situation evolved. And that's not because people are short-sighted; it's because the combination of broad, fuzzily-defined goals plus a sensitivity to environment allows you to improvise successfully — and turns out to be a very adaptive and successful way of dealing with the world. (For heavy duty philosophical background to this, see Hubert Dreyfus' criticism of the concepts underpinning expert systems.)

If you assume that tutors operate the same way then you end up with a picture of 'teaching as performance' with an emphasis on a process that is always up for negotiation and adaptation by the participants (that is, learners and tutor).

Can LAMS be developed further to allow this improvisation and adaptation of learning sequences after learners have begun the sequence? It might not be straightforward, but if it could be done, LAMS would have a real advantage over many other e-learning materials, which — notwithstanding the claims they may make about personalisation — tend to follow pre-programmed paths.

And going back to Seb's comments on the attrition rate in the evaluation study, what chance of the funding and evaluation regimes for e-learning pilot projects being sufficiently flexible to adapt to evolving circumstances like the software being not quite ready or the pilot participants not following through to use it extensively? This isn't a criticism of the report's authors, who have produced a valuable document in what seem like difficult circumstances, but it looks as though when the evaluation was planned it was expected that the pilot experience would be different from how it turned out.

Posted by David Jennings in section(s) E-learning, Human-Computer Interaction, Teaching on 19 June 02005 | TrackBack
Comments

Interesting perspectives. I agree to some extent with the implied criticism that LAMS presently lacks flexibility in f2f use -- but so, I would guess, does the average lecture! However, LAMS does have some activities that enable addition of content on-the-fly if you plan ahead to do this (!) and I found it sufficiently easy to use that I have hacked a sequence in class in response to unforeseen problems. It's just that students then have to skip into the modified sequence -- a bit of a pain. As to sequences being too linear, I suspect that's due to an excessively long sequence and, probably, an instructivist approach on the part of the author (mea culpa), not necessarily a failing of the software. Having said that, many teachers are likely to fall into that trap and one real benefit of LAMS is that it will hopefully encourage greater reflection on the part of teachers. The next version will apparently also have wizards to assist with design if required.

Posted by: Peter Miller on 5 July 02005 at 12:38 PM

James Dalziel of the LAMS Foundation tried to post a response to this article about ten days ago. Unfortunately the comments script on this server was suffering one of its occasional periods of downtime (thanks to the extreme loads caused comment spam), and I was away on holiday and unable to deal with the problem. But, belatedly and with apologies to James, here is his response, plus a comment of my own at the end.

David
-----------

When we started building LAMS, we had always hoped that it would be possible to edit a design "on the fly" to allow teachers to adapt to changing class circumstances, or new realisations about how to proceed that only occur once you're in the thick of it (the need to change a lesson plan mid-stream is familiar to all classroom teachers).

(You can even guess where this feature would have been in the LAMS interface - under the second tab in Monitor, where the visual sequence depiction from Authoring is repeated for the live class).

However, LAMS was a very difficult piece of software to build, and one of the casualties during development was the ability to edit on the fly - it just proved too hard in our first build to make the software so flexible that it could be changed mid-stream without making the entire application too unstable.

So this feature didn't make it into V1.0 - and while there are many, many features I wish LAMS had today, this is one of the most fundamental that is missing (in my view), not just because it would be a cool feature, but because of its pedagogical importance. But the reason it didn't make it was sheer software difficulty, not an assumption that teachers *shouldn't* change a design on the fly.

(As Peter notes, there are some options for changing a running sequence in LAMS today through the "Define in Montor" feature - which allows some tools to change their internal text (eg, the text of a question in Q&A, the voting categories in Voting, etc). But I agree with Peter that you need to "plan" to use this feature ahead of time to make it work.)

The good news is we've been working hard on a complete rebuild of the backend of LAMS for the past 9 months, and hope to have this out in beta in the second half of this year. This version (V1.1) will allow you to edit most aspects of each tool even when a sequence is running (ie, more "Define in Monitor" options).

Also, we're planning to implement full editing of sequences on the fly in the next release (V1.2), so you could start a sequence with just a noticeboard, and then design everything else on the fly. This should go a long way to replicating the flexibility of f2f teaching. The only contstraint will be that you can't edit a task once at least one student has started it - you can only edit tasks than no-one has got to yet (you can use the "Stop" feature to help you control this where needed). One nice byproduct of this new feature is that when a teacher does build a lesson on the fly, at the end you have a record of the sequence, so if it worked well, you could then save it in authoring, and run it again or share it with colleagues.

As to whether teachers can use a planning tool like LAMS and think several steps ahead, my experience of watching LAMS trials is that this is certainly possible - perhaps because lesson planning is part of everyday practice for many teachers, so that once they have basic familiarity with the software, they can transfer this skill across to LAMS.

For what it's worth, I don't think the photocopier example really captures what we're seeing in the field with LAMS; and personally I wouldn't draw the analogy from software engineering with end users to LAMS lesson planning. But it will be interesting to see how further research informs this debate, and I appreciate the suggestions from HCI on how we might understand this new area.

James Dalziel
-----------

I wasn't meaning to say that lesson planning is an activity analogous to a software engineer designing a software program (or indeed to a photocopier user planning how to get the copies they want). The critique from Suchman, Dreyfus and others is actually more general, and more profound than that. It doesn't depend on characterising or comparing particular types of activity; it argues that all human activity has an evolving, improvised, in-the-moment nature.

David

Posted by: David Jennings on 18 July 02005 at 10:12 AM
Post a comment









Remember personal info?