Jonathan Stevens, School of Theatre, Performance and Cultural Policy Studies & Mark Childs, Centre for Academic & Professional Development, University of Warwick and Erik Lint & Peter Eversmann of University of Amsterdam, The Netherlands
“I was still daunted about finding contemporary relevance for a piece largely about outdated shipping laws in a small Dutch village at the turn of the last century.”
(Lee Hall, The Good Hope, vi)
The JISC/SURF funded, ‘Streaming Theatre’ project centres around the Cross-Cultural Performance Analysis module, which is taught jointly between the School of Theatre Studies at the University of Warwick and the Theaterwetenschap at the Universiteit van Amsterdam (UvA). This article explores the challenges faced between the partner institutions in devising adequate systems for student evaluation and assessment throughout the project. Initially focussing on the appropriation of formative platforms that best-fit the module’s learning outcomes, we move to evaluate the systems in place (both human and technological) that feature in the final summative assessment. In conclusion, this article addresses key issues that will inform the future development of assessment methodologies within the Cross-Cultural Performance Analysis module. The primary argument is that the process of establishing, then enforcing shared responsibility while simultaneously building confidence with technologies should form the bedrock of distributed learning environments and ongoing assessment within them.
The role of evaluative reflexivity within this process is three-fold. Firstly as part of an action evaluation cycle, evaluative reflexivity engages all of the participants in a study in the outcomes of that study. By asking them to reflect on their aims, goals and benchmarks of success, the evaluator creates “buy-in” amongst the participants in the practice being developed (Conradi, 1998). Secondly, the process of reflecting within the development cycle enables the evaluation to be formative rather than merely summative, and hence improve the focus of the project. Thirdly, and perhaps most importantly within a learning and teaching context, encouraging students to take part in evaluative reflexivity promotes their ability to display meta-cognition and self-criticism (Bing, 1997).
The Streaming Theatre module was devised by Prof. Peter Eversmann (UvA) and Prof. Jim Davis (Warwick) as a way to introduce undergraduate students in Theatre Studies at both universities to the differences between theatre traditions and current theatre practices in the Netherlands and the UK. The collaboration builds on existing research links between UvA’s Theaterwetenschap and Warwick’s School of Theatre and Performance Studies, the DIVERSE nework  and experts from the JISC/SURF-funded ‘Stream Team Experts Group’ .
Objectives for new modes of assessment
As the flagship project for the Stream Team Experts Group, the project is necessarily experimental and appropriative in scope. In order to address the diverse teaching and learning objectives for the project (see table 1), experts from Warwick and UvA devised a programme that embedded several learning technologies into the course curriculum. No technology was employed without a perceived need, yet it emerged that several technologies would need to be employed in conjunction so as to effectively conduct summative assessment.
Table 1, Formative Assessment Factors
|Students post personal introductions and photographs on group blog.
|Introduce UK and NL student cohort and establish familiarity between groups.
|Offline seminars and readings
|Introduce students to notions of cross cultural performance, cultural embeddedness and ‘postdramatic theatre’.
|Warwick Blogs / Virtual seminars (Macromedia Breeze)
|Students post examples of ‘typically Dutch’ and ‘typically British’ objects, people and practices then continue discussion from blog in seminars.
|The core concepts are addressed through concrete examples found by and discussed with their peers while the exchanges are monitored and steered by faculty.
|Streaming video (Windows media streaming server)
|Expose students to a range of performances in both English and Dutch versions.
|Students discuss content from offline seminars and readings with wider (cross-cultural) peergroup,
|Students’ understanding of and ability to employ diverse theory is tested while being informally assessed by faculty.
|Students view streaming video of performances
|Students comprehension of performances in the context of CCPA module is steered towards appropriate level/outcome.
|Warwick Blogs and Forums
|Students are assigned reflective tasks on core concepts and asked to source ‘original’ material.
|Faculty assess and steer students’ independent skills towards appropriate level for future individual work.
|Small group work
|Warwick Forums and Instant Messaging technologies
|Students are paired off (1 UK, 1 NL student) to devise presentations on the ‘same’ performance in English and Dutch version.
|Students’ given opportunity to prepare presentation for group preparation, which is appraised by faculty in preparation for individual work.
|Small group presentations
|Macromedia Breeze / Powerpoint
|Students present joint presentation.
|More formal advice and feedback given to students prior to summative assessment.
|Online lectures (Breeze and Powerpoint)
|UK and NL faculty lecture on respective theatre traditions.
|Formal lectures with Q&A sessions which enable qualitative assessment of student comprehension.
Despite apparently implementing every available learning technology during course delivery, due to the unique challenge of teaching, mentoring and ultimately assessing the distributed cohort; each technology demonstrably served a bespoke function.
Engaging students in collaboration
Without establishing familiar links and concerns, the students’ participation in seminars may have been less fruitful. While, in an article on the theme of assessment, this comment may seem incongruous, the responsibility of faculty in the UK and NL was to ensure that students were comfortable and capable of information exchange prior to indulging in the course’s more rigorous critical content.
The following extracts from student blog entries show the beginnings of exchanges that enabled the students to work collaboratively as well as individually within an intellectual framework supported by their shared experiences as a group:
Ps1: I like about Englad: [sic]
- the Brittish accent
- the humor
- my Brittish relatives
- Ewan Mcgregor!
Ps2: I like less about England:
- the Queen
- the food
- the absurd time they close pubs!
(<Student Name>, Streaming Theatres Blog, 23rd October 2006)“I chose this module because I want to find out whether people who study theatre in other countries are subject to the same stereotyping as we are.”
(Hello, Streaming Theatres Blog, 29th October 2006)
The blog enabled staff to address emergent concerns. For example, responding to the first comment, faculty took time to disaggregate the notions of ‘England’, the ‘United Kingdom’ and ‘Britain’ (Northern Ireland, Scotland, Wales, the ‘North/South divide’ etc). As English is the lingua franca of the Cross-Cultural Perfomance Analysis module, the NL students logically associate English with ‘England’. The apparently innocuous comment about Ewan McGregor (a Scotish National) on the blog prompted staff to cover the regional and national particularities of the UK during the virtual seminars and stress the differences in arts policy and theatre traditions within its constituent states.
Secure Warwick Forums were created for the seven pairs of students (one UK and one NL) working on joint presentations. Forums provide a space that serves as a notepad for student exchanges as they develop presentations. Staff are able to assess the ongoing work on the forums and steer the collaborations towards an appropriate level in terms of content and register. The forums also provide an audit trail that enables faculty to formally address issues of non-participation, dispute and unbalanced input between the pairs.
Between the 7th and 29th of November 2006, one student failed to reply to their counterpart’s seven entreaties to begin discussing the topic of their presentation and to devise its structure. The student attempting to make contact grew increasingly distressed and was able to demonstrate that they had not been responsible for the delay in starting. Staff were then able to contact the uncommunicative student and devise a timeframe as well as a management strategy for the students to complete the joint assignment. Without the ability to assess the groups’ progress on the Forums, the task of mandating the student to complete specific tasks would have been less transparent. Furthermore, due to the distributed nature of the project, the Forums provided an ideal platform to monitor ongoing work with a minimal expenditure of effort.
Under their own initiative, six of the seven pairs working towards joint presentations exchanged personal Instant Messaging details (Microsoft Messenger and Skype), then used these proprietary platforms to collaborate.
Tutor moderation of students’ online activities
The entrepreneurial nature of the students’ initiative is commendable, yet the convenors at Warwick and UvA shared concerns that unmonitored exchanges could detrimentally affect their ability to conduct ongoing evaluation and assessment. The students readily accepted the argument that logs of their exchanges should be kept for the aforementioned reasons (steering content and auditing) and uploaded logs of their work to their individual forums. In future collaborations such diligent recordkeeping may not be necessary as there were substantial tracts of anecdotal and social chatter amongst the material that served staff in conducting their evaluation.
Even so, several of the chat-logs possess qualities that enabled staff to acquaint themselves with the student experience. The value of these records in a pilot such as the Streaming Theatre project should not be underestimated given that the novel pedagogies are untested in context. Arguably, staff managing any distributed collaboration should endeavour to assess ostensibly peripheral information even if it only tangentially affects the outcome of their project.
[I]s Shakespeare something that is often adapted in Holland?
Dutch people like Shakespeare, but in England I assume that Shakespeare is much bigger because he's the national pride of England (or am I wrong?)[...] Can you compare their approach to the one of Mafalaani?
I believe that Dutch and English forms of staging can be fairly similar – the use of light/projection and bare, 'nowhere' spaces. This is true of a lot of English Shakespeare productions.
Yes, we should keep that in mind as a point of similarity!
(Warwick Forums, 20th November 2006)
Integration with classroom seminar work
The primary goal during the weekly ‘virtual seminars’ was establishing an effective exchange relating to appropriate course content, but the seminars also served the purpose of establishing a permissive, collegial atmosphere in conjunction with the first blog-based activity. Reiterating the statement that formative assessment may conventionally be centred on evaluation of knowledge, critical skills and appropriate feedback, faculty used the opportunity afforded by the telematic exchanges to explore the development of students’ technical competencies, their ability to cooperate in a seminar where half of the participants are in a foreign country as well as the students ability to communicate across cultural and linguistic barriers.
Evaluation of assessment models
The course convenors at Warwick and UvA held weekly videoconferences outside of the virtual seminars in order to assess the students’ progress in all the above capabilities. Recommendations for improvement included:
The UK students vocabulary outstretched that of the NL students and the speed at which they spoke caused frequent misunderstandings. Warwick students were asked to consider their audience carefully, avoid vernacular, hasty speech and take time to look at the reactions from the NL students.
- The UK students began their term three weeks after the NL semester, it was evident that their familiarity with core theory was lacking. Warwick staff increased frequency of offline sessions in order to redress the imbalance.
- There were various ‘sign-languages’ developing that ameliorated the difficulties caused by technical errors, but they differed between UK and NL. Staff implemented a standard set of visual signs including hands cupped to ears to denote loss of audio.
Before moving to evaluate the summative assessment of the project from the Warwick perspective, it should be noted that the UvA students undergo constant evaluation. Staff at both institutions had to accommodate the different institutional assessment schemes. If, for example, Warwick students failed to reply to a question from UvA, the student at UvA may be detrimentally affected while the Warwick student’s grading is not (currently) reliant on their participation.
The absolute necessity to accommodate such difference is best summed up in reference to a particularly capable student who tended to take charge during seminars (seizing the microphone, for example). In order to ensure all UvA students had the opportunity to achieve their potential, the student was simply asked to ‘reel it in’. Although the step had to be taken, Warwick and UvA staff consulted each other before taking the action in order that it didn’t negatively affect the ability of either side of the cohort to function effectively during the seminars.
UvA’s Theaterwetenschap typically models student assessment on an ongoing scheme. Warwick’s School of Theatre Studies adopts two schemes of assessment dependant on the nature of modules. Where students follow conventional lecture/seminar-based modules, assessment features two 3,000 word essays over the Christmas and Easter vacations and one three hour exam during the summer term. The second scheme is designed for practice-based courses and features ongoing assessment of contributions to group work and interim projects such as short essays and performances. As the Cross-Cultural Performance Analysis course had been approved by Warwick’s Board of Undergraduate Studies as ‘conventional’ and the scheme of assessment had been communicated to the students opting for the module, it was not possible to remodel the assessment in time for the beginning of the course.
This caused a certain degree of conflict between the partners as the students at UvA were assessed on each task set as well as their contribution to the seminars. Faculty at Warwick communicated the need to contribute consistently and expansively to the UvA students’ Blogs, Forums and Seminars, however this placed staff in an invidious position given that their ability to mandate students to contribute was limited. The initial team-building blogs coupled with the keenness of most Warwick students to contribute more than the bare minimum required of them resulted in a productive exchange, however, in the future, it will be necessary to establish a scheme that weighs some of the Warwick students’ grade towards their contributions during the course as well as to the summative assessment tasks.
The essay component of the Cross-Cultural Performance Analysis module was to be a conventional paper-based essay at Warwick while UvA students were to produce ‘pictorial essays’. Pictorial essays, are sub-sites within the i-theatre website hosted by UvA. They are individually composed by students and feature embedded fragments of streaming video. Developing pictorial essays is a critical learning outcome within the Cross-Cultural Performance Analysis module:[Students] have to carefully analyse and select the dramatic turning points. By doing so they learn how to establish the different ‘signatures’ of the theatre makers. […] how to write for the web and how to select, cite and hyperlink the audiovisual performance fragments.
(F. Lint and P.G.F. Eversmann, Streaming Theatre)
The argument that the pictorial essay should replace a ‘conventional’ essay was sufficiently compelling for the Chair of Theatre and Performance Studies at Warwick to approve its use for summative assessment. The pictorial essays will be assessed using the standard scheme for undergraduate students in the Humanities at Warwick, however there are three considerations that will inform on their grading.First, as the pictorial essays are websites, the students have the option of ‘writing for the web’. The question of whether or not the essay is appropriate within the medium is addressed in two phases. If the student has composed an essay that is a ‘typical’ undergraduate paper in response to a defined question, this will not be perceived in a pejorative sense by the markers. Concurrently, if a student has addressed the essay in a manner that is less ‘typically academic’, for example, by lowering their register and limiting their vocabulary, this will not negatively affect their grade provided the essay represents a level of critical thinking and a conscious decision to produce a piece that is appealing, as well as accessible, to a wider audience.
This segues into the second consideration regarding ‘matching critical skills’. Pictorial essays feature video fragments, hyperlinks and images in line with text. If students demonstrate a sound understanding of the media used to compose and complement their arguments through their pictorial essays, then these factors should be considered when assigning grades. Once again, however, students who have not hyperlinked, embedded fragments of video and images into their essays, should not find that their results are negatively affected provided there is a contextual justification for their decisions.
Supporting the students’ learning experience
Finally, ‘has the student received the necessary support?’ Should it emerge that a student has not understood the technology, the task, or the nature of the medium, it will be necessary to devise strategies through which to ensure that the student is not ‘marked down’. As a pilot, the Streaming Theatre project has taken certain risks, yet in teaching a live cohort of students at honours level, faculty at Warwick and UvA need to be reflexive about their own performance and swift to react should it seem that their students are not capable of producing the work required.
Despite the fact that students at Warwick are not graded until the summative assignments, their contributions to seminars, blogs, forums, instant messaging and email exchanges is pivotal to the success of the module and their leaning. Through frequent review meetings and immediate responses to concerns raised by students and staff at partner institutions, it was possible to build a collegial atmosphere where responsibility was shared and accepted. By continually assessing the structure of the project, the procedures and technologies involved; the partners were able to proactively incorporate change. Reflexive evaluation must be embedded into the process of course delivery and ongoing review in distributed learning environments.
As the module moves towards its second year, a more transparent assessment scheme will be in place that is identical at both institutions. An interim advisement to individuals interested in establishing telematic collaborations between any partners is that all schemes of assessment should be devised once the content of the course has been finalised. Furthermore, despite the corollaries between assessing telematic engagement and classroom/seminar contribution, any assessment of formative work should take into account the differences between media employed (blogs, forums, extra-mural messaging, video-conferencing and so forth).
 “developing alternative modes of learning and teaching to complement the asynchronous text-based formats that predominate within e-learning“ More information about the DIVERSE network is available here: http://www.diverse-video.net/
 More information about the JISC/SURF supported Stream Team Experts Group is available here: http://video.surfnet.nl/info/streamteam/streamteam.jsp
Bing, D. (1997) Action-Evaluation Progress: Lessons Learned Over the Past Five Months. Paper presented at National Conference on Peacemaking and Conflict Resolution. Pittsburgh,
Conradi, L. (1998) A Comparative Analysis of Action-Evaluation Across Multiple Cases, http://ariagroup.com/AE_essays/docs/AE_Comparative%20Analysis.pdf, accessed 30th April, 2007
Hujermans, Herman (edited and translated by Hall, Lee); The Good Hope; London: Methuen Drama; 2001.
Lint, F., Eversmann, P.G.F and Childs, M.; Streaming Theatre, Project proposal for JISC-SURF streaming media pilots 2005-2006; Amsterdam; 2005. Available for download from “Streamteam Pilot – Streaming Theatre”: http://video.surfnet.nl/info/streamteam/artikel_content.jsp?objectnumber=99010 (Last accessed, 4th February 2007)
Lint, F.; i-Theatre – community for performance and drama education; Netherlands: Amsterdam, 2006
URL: http://www.i-theatre.net/ (last accessed 4th February 2007)
Stevens, J.; Streaming Theatres; University of Warwick: UK; 2006.
URL: http://www.warwick.ac.uk/go/streamingtheatres (Last accessed 4th February 2007)
Streaming Theatre Students and Staff; Streaming Theatres Blog; University of Warwick: UK; 2006.
URL: http://blogs.warwick.ac.uk/streamingtheatres (Last accessed 4th February 2007)
Streaming Theatres Students and Staff; Forums from the Streaming Theatres Module; University of Warwick: UK; 2006.
URL: http://forums.warwick.ac.uk/wf/browse/category.jsp?cat=592 (Last accessed 4th February 2007)
N.B. Warwick Blogs, Warwick Forums, the Streaming Theatres website and the i-Theatre website are all currently secured pending the completion of student assignments and grading. Exemplar material from these resources will be made available from May 2007 through the i-Theatre and Streaming Theatres websites.
Citation for this Article
Stevens, J. & Childs, M. & Lint, E. & Eversham, P. (2007) Assessing the stream: Evaluative reflexivity in a multi-platform e-learning project Warwick Interactions Journal 29. Accessed online at: http://www2.warwick.ac.uk/services/cap/resources/pubs/interactions/current/abstevens/stevens