The Signpost

Special report

A post-mortem on the Indian Education Program pilot

Synopsis

Initiated in the wake of last year's successful Public Policy Initiative, the India Education Program is the latest culmination of the Wikimedia Foundation's stated strategic goals of university outreach and expansion into the Global South. It quickly ran into difficulty, however, when it emerged that a significant proportion of the articles submitted by students in the program failed to show an adequate knowledge of basic editing skills and did not respect Wikipedia's copyright and plagiarism policies. Many students seemed manifestly unfamiliar with rudimentary wiki syntax and fundamental competencies such as sandbox creation and responding to talk page messages, and a minority appeared to lack the English language skills requisite for productive interaction. Furthermore, many of the topics assigned for editing by students were in technical areas traditionally well-served by Wikipedia's native editing community, which made for a difficult time for comparatively inexperienced students trying to improve advanced content. These issues were exacerbated by the impositions of regular academic deadlines that put pressure on students to submit something for fear of failing classes.

Frank Schulenburg, Global Education Program Director for the Wikimedia Foundation, who described himself as "deeply frustrated" with the outcome of the program.

As the talkpage of the project shows, the resulting cleanup efforts, including a contributor copyright investigation (CCI), generated much dissatisfaction and questioning of the preparations and management on the Wikimedia Foundation's end. Disgruntled administrators and new page patrollers accused the program leadership of failing to anticipate and pre-empt the enormity of the maintenance task facing community volunteers, and the entrenched cultural differences in understanding of plagiarism between the editing community and the students. They also questioned how it was appropriate for what was ostensibly a pilot program to involve more than eight hundred students. In response, Nitika Tandon, the consultant directly overseeing the project, acknowledged the copyright concerns and outlined the steps that had been taken to address them, including in-class lessons on copyright, requiring students to submit their work in sandboxes prior to ambassador or professorial approval, and one-on-one counselling for offending students. IRC office hours with the IEP team were held on October 12 and October 21. The copyright violations continued unabated however, and some editors went so far as to question why the community was tolerating such an initiative, even suggesting the program should be brought before the Arbitration Committee prior to any future activity.

On October 18, in the light of the continuing onslaught of copyright violations, arbitrator emeritus Wizardman called for the project to be unilaterally shut down, citing it as a net negative for the encyclopaedia. In the ensuing discussion, Kudpung, who had been leading the cleanup efforts, highlighted two unwelcome impacts upon the Wikimedia movement he called "blatantly obvious" but which seemed "to have been totally disregarded" – that students would not be attracted to edit Wikipedia by being put through a stress-inducing deadline-tied program led by highly inexperienced ambassadors, and that the community's administrators and new page patrollers had been demoralised by having been forced to deal with cleanup of problems not of its making. After a further week of problems, Calliopejen1 proposed a wholesale removal of all unsourced text contributed for the project. On November 3, following a meeting with the Director at Pune's College of Engineering, one of the participating institutions, Tandon announced the decision to call for an immediate end to the students' editing of Wikipedia, and for a one-month moratorium on student contributions while the backlog of copyright investigations continued. Students who had added good quality content would be rewarded with marks based on the quality of their edits, those who had either added plagiarized material or had not even started would lose marks, and anyone who continued editing would receive negative marks. As 13 out of 14 classes at the Symbiosis College of Economics had already concluded, only two classes remained active in the program at the time; one each at Symbiosis School of Economics and SNDT Women's University.

Apologising for the massive cleanup effort presented by the initiative, Wikimedia Foundation's Global Education Program Director Frank Schulenburg expressed 'deep frustration' with the outcome of the program. He admitted fault on the organisers' part in the delay of getting online ambassadors working with the students, an inadequate articulation of the ambassadors' role, and poor communication on the foundation's part, but tried to dispel the notion that any particular group was primarily culpable for the difficulties. Copyright specialist administrator Moonriddengirl, who is the current community liaison for the Wikimedia Foundation, expressed sympathy with beleaguered volunteers exasperated by the perceived lack of assistance from the foundation, but emphasised that staff were handcuffed from intervening officially by the foundation's Section 230-determined legal imperatives not to act as a publisher of content. Nitika Tandon has prepared a draft summary of "findings and learnings" from the program.

The administrator's perspective

The New Page Patrol backlog for the month from October 4 to November 4, cited by Kudpung as illustrating the herculean efforts of volunteer patrollers in addressing the flood of submissions.

The Signpost asked admin Kudpung for his reflections on the experience of dealing with the administrative headache generated by the program:

"My involvement began when I blocked the IP address of one of the faculties in an attempt to stem the massive flow of copyright violations. It was only after my curiosity took me deeper into the issues, that I became fully aware of the scale of the problem. My talk page became for a while the hub of communication – a situation that should never have arisen, but it was very difficult to know who was in charge of various parts of the project and whom to address. I've been teaching here in Asia for many years, and I was also concerned that during the planning stages the American side of the operation may not have taken the challenges of the cultural dichotomy into consideration. I spoke with several members of the US WMF staff in an endeavour to learn if they were aware of the extent of the issues and if anything of consequence was being done. I also spoke with Hisham several times to obtain some reassurance on behalf of the community that something would be undertaken at ground zero, and finally the organisers held discussions in the USA. Some of the things I posted on various talk pages may have been perceived by some as accusatory in tone, but I felt it was necessary that people be galvanised into action."

"I understand the importance of Wikipedia reaching out to other regions, especially to those like India that have strong ties to the English language, but the lesson drawn from this pilot project is that things work very differently in other countries, and careful, long-term preparation with the involvement of the community is essential."

"For future extensions of the project, it is paramount that the Indian Campus Ambassadors are more accurately selected and trained, and have an adequate working knowledge of editing and basic policies. It is equally imperative that new editors can benefit from user friendly page creation tools, and that a much improved system for the control of new pages along with a replacement for CorenSearchBot are made available to page patrollers as soon as possible – the next wave of Indian students and their ambassadors is going to need them. Ironically, if WP:ACTRIAL [Ed. A local initiative to restrict page creation to autoconfirmed editors that the Foundation chose not to implement] had been implemented, it would have spared the students much of their disappointment and embarrassment, and the maintenance community much of the stress they volunteered to be subjected to. Nevertheless, from what I have seen of the new tools already, I am optimistic that the WMF's core philosophies and outreach programs can be further developed and maintained."

The consultant's view

Hisham Mundol, a consultant for the Wikimedia Foundation with responsibility for the India Programs, on day 2 of the Campus Ambassador training in Pune.

The Signpost interviewed Hisham Mundol, consultant for the Wikimedia Foundation's India Programs, to get an insight into his perspective on the situation and what it might mean for the future of such initiatives. A transcript of the interview follows.

The Signpost: What was the planning process behind the India Education Program, what was the involvement of the different roles (e.g. contractors, foundation staff with prior experience with university outreach, foundation management, Wikipedians, professors, ambassadors), and how did it differ from previous university outreach programs (such as the Public Policy Initiative in the United States)?

Hisham: The Strategy Project, which involved more than one thousand contributors to Wikimedia projects, identified increasing participation in a number of countries as a priority – and India is the largest of these. I head up a team that's tasked with catalyzing this agenda. Given the demographic profile of India (i.e., a relatively young population) and the context of students, an education pilot was planned in India.

The India Education Program pilot grew out of the success of the Public Policy Initiative in the United States during the 2010-11 academic year. I worked closely with WMF Global Education Program Director Frank Schulenburg, Global Education Program Manager Annie Lin and US Campus Ambassador PJ Tabit on designing the pilot. One decision we took early on was to focus the pilot in one city (Pune) because we knew from the U.S. pilot that it was easier to focus on one city. (Pune has a strong heritage for education in India.)

Frank, Annie and a professor from the University of Mississippi named Bob Cummings, who all had experience with Wikipedia in the U.S. higher education system, traveled to Pune, India, to kick off the pilot. PJ stayed over for just under 3 months in Pune to help establish the pilot.

Pune's campus ambassadors in a lighter moment

Frank, Annie, and I personally went through 700 applications from people who wanted to be Campus Ambassadors in India, and we chose the best 20 candidates based on their understanding of Wikipedia (even if nascent), their ability to learn, their ability to teach, their commitment and their motivators. As in the US pilot, many had never contributed significant amounts of content to Wikipedia, but all were eager to contribute in other ways - such as teaching students how to edit. (A learning from the US pilot was that Campus Ambassadors with little prior Wikipedia-editing experience did just as well as – and sometimes better than – long-term Wikipedians when it came to performance as Campus Ambassadors, because their role is to introduce students to the basics of Wikipedia on a face-to-face level.)

I also hired Nitika Tandon to focus specifically on the India Education Program, as my role also includes other responsibilities in India.

We had a team of Online Ambassadors (most of whom were experienced Wikipedians) but a failing from the India pilot is that we were unable to bring them on-board early in the semester.

What note did foundation staff take of cultural attitudes specific to India regarding attribution, copyright, responsibility and co-operation?

Nitika and I were both born and live in India. We have an intimate understanding of India cultural attitudes. I personally don't believe that Indian culture had much bearing on this pilot. Some students in India – as elsewhere – are either lazy and plagiarize or they genuinely believe that close paraphrasing means something is no longer plagiarized. Some students in India - as newbies from elsewhere where English is not the first language – find it challenging to write in language appropriate to Wikipedia. And, some students in India – as newbies elsewhere – value responsibility and co-operation but are unfamiliar with Wikipedia culture and were unresponsive to actions & comments from others in the community taken in reaction to their edits or comments left on their talk pages.

Having said this, the real challenges were not cultural but programmatic. It's a pilot, and we've learned many important lessons. There were two that were most important, though. First, engage the community early. When the power of the community is brought to bear on a challenge, we get a huge number of solid ideas and a tremendous amount of help. It follows the open-source adage that "given enough eyes, all bugs are shallow." Second, ensure that all the infrastructure elements are in place and adequate (e.g., Campus & Online Ambassadors), selection of colleges and faculty based on involvement and capability, modifications in training of Ambassadors, faculty and students.

Do you feel you had the support, background briefing, and relationship with Wikipedian volunteers needed to address the issues that arose with the student submissions? What might have made your job(s) more effective?

I think communication with the global Wikipedian volunteer community was amongst the biggest failings of the pilot. While we had invested time and effort with the local community, we can and should have done so much better with and for the global community.

Another aspect, as I mentioned earlier, was that we had a team of Online Ambassadors to provide that crucial on-wiki role of assisting students – but, unfortunately, recruiting and training new India-focused Online Ambassadors took time, and it simply didn't happen fast enough.

We've had a lot of good interactions with the Wikipedia community through this process – we've had two IRC office hours in which we were able to explain our thinking to several editors. And we were blown away when we asked the U.S.-focused Online Ambassadors to help out with the India program cleanup too – more than 20 of them immediately offered to take on extra duties. We're so thankful that we have such a wonderful community of people who love Wikipedia as much as we do.

One point I'd like to emphasize here is that the India Education Program is a pilot program. In pilot programs, you start small (we should have started much, much smaller!), and you try different things to learn what works and what doesn't work. We adapted what had and hadn't worked from the U.S. pilot, but India is a different place, and we had a whole host of issues that the U.S. program didn't have (and we didn't have some problems the U.S. pilot had). The nature of pilot programs is to learn from them, and we've certainly learned a lot from our pilot.

When did it become apparent that there were widespread serious issues with the submissions from students in the program, and what steps or decisions did Foundation staff take in response?

We realized that at the beginning of September when a community member alerted us to the problem. Immediately, we dispatched our Campus Ambassadors to go into the classrooms and teach students about copyright violations and how to avoid doing them; Campus Ambassadors conducted more than 20 in-class sessions about copyright. Nitika and I also conducted about 15 in-class sessions with the faculty members. We pulled up students' copyright violations on the screen and showed them why this was a copyright violation and how they could fix it. Campus Ambassadors reached out to students desk-by-desk in class, by email, by text, by Facebook messages, and any other way they could think of to encourage students to stop adding copyrighted materials to Wikipedia. Our Campus Ambassadors poured their hearts and souls into telling, directing, coaxing, cajoling and begging students to not add copyvios to Wikipedia, but some students simply would not or could not understand.

By early October, we concluded that some students just weren't getting it, no matter how hard or how often we tried. We instructed all students to stop editing directly in the article namespace on Wikipedia; instead, we encouraged them to only edit in sandboxes. While we did not want students to add copyrighted materials to sandboxes either, we wanted to provide a way for us to check the students' work before it went live on Wikipedia. Many students still continued to edit in the live article namespace. Therefore, on 3 November, Nitika and I went into the College of Engineering at Pune (at Symbiosis School of Economics, deadlines had already passed for all but one class so students had stopped editing) and met with the director, who shut down the program and told students they would not be graded on anything they had added to Wikipedia after that date. The students who added good quality content would be rewarded with marks based on the quality of their edits and those who had either added plagiarized material or had not even started would lose these marks. To ensure that the problem was adequately controlled and that students actually stopped editing, the Director also stated that anyone who continues to edit will be given negative marks. (This last point is only for the remaining part of this semester.)

Did staff members feel restricted in responding to the issues by the legal/policy imperative (of the Foundation as a service provider rather than publisher) not to directly address content?

Yes. Of course we do. It breaks our hearts to see copyvios in Wikipedia text and all of us – both in India and back at the Foundation office in San Francisco – want desperately to go in and take them out ourselves, and to join in the large-scale cleanup efforts. Unfortunately, the best advice of our legal team is that we shouldn't do that, because it would be interfering in the content creation role, and could compromise our "safe harbor" immunity. These are constraints that we abide by as a result of working with / for WMF. This is another reason why we are so grateful that members of the community have volunteered to carefully check and correct the work of students.

How would you respond to the idea that it was not clear to Wikipedians who attempted to resolve the issues who was managing the project? And to their frustration as problematic submissions continued?

I'd say that from their perspective, they're probably right. We could have dealt with that using better communication processes. I'm really sorry that we didn't communicate more. This is no excuse but we spend all our time meeting with professors, Ambassadors, and students, trying to resolve issues. I wish we had communicated more with the global Wikipedia community. There were intensive efforts and communications that occurred but these were shared within the limited group of Ambassadors, faculty and students. We should have put out a lot more information on these much earlier. For instance, we could have put every important email sent by us to Ambassadors or from Ambassadors to faculty or students on public wiki pages.

I can understand the frustrations of community members. I share their frustration that some students continue to submit sub-standard material. This was one of the reasons why we decided to suspend the program at the College of Engineering Pune.

From the bottom of my heart, I thank every editor who has helped us address the quality issues coming from the India Education Program.

Do you have any personal thoughts to share on the project, or what it has revealed about the nature of university outreach and Foundation/editing community relations?

I love Wikipedia. I believe this program has considerable potential – if managed effectively – to promote participation and expand the community.

We've made mistakes in this pilot – and we've learned a great deal. While it's the nature of pilots for there to be challenges, there are some we can and should have avoided.

I also want all the issues that we have had not to cloud the wonderful contributions of so many students. We have a bunch of students who a few months ago had never edited Wikipedia, indeed didn't even know that they could edit or how to. Today, there are some remarkable contributions by them. I hope many of these students will become prolific editors going forward.

Conclusion

In light of the Foundation's significant resources, its prior experience of university outreach and the enthusiasm of applicants to become Campus Ambassadors, the failure to adequately prepare students for the task that faced them is troubling. It is not clear, for instance, why exactly the willing input of Online Ambassadors was not incorporated into the program at an early stage. One perhaps compelling explanation could lie in the hypothesis that local conceptions of copyright and attribution were so distinct from those expected on Wikipedia and so entrenched that ambassadorial evangelism could have little effect in the short term. However, Mr. Mundol’s rejection of this hypothesis makes it difficult to explain the wild discrepancy of outcomes of the IEP and the Public Policy Initiative, which only finished up in September. Yet more worrying is Mr. Mundol’s conclusion that the dramatic failings of the project are attributable to programmatic mistakes – a claim which casts doubt on Mr. Schulenberg’s position that there is no single nexus of culpability for the problems which arose and that will not fill the community’s hearts with confidence in the Foundation’s due diligence in project management. The staff members involved can be excused for not yet agreeing upon a clear determination of just what went wrong as the program winds down and focus appropriately remains on the not insignificant cleanup task, but arriving at such an analysis must surely be a strategic priority in the coming weeks.

The Wikimedia Foundation is a young and fast-growing organisation. It is pursuing with intensity laudable and ambitious strategic goals derived from an innovative and volunteer-respectful consultation process. Like its projects, it has pursued these goals with a healthy attitude towards risk, not prone to overcautiousness where gains may be made. The openness to criticism and oft-voiced declarations of appreciation for volunteer assistance of Foundation staff is commendable. This experience of the Indian Education Program, while a setback, should not result in a retreat to conservatism for outreach and expansion efforts.

Yet it is important that the Foundation give due consideration to thorough research and planning in preparation of its initiatives, and ensure these do not take for granted the patience and indulgence of the volunteer community. Failure on the former front would be to squander donor resources; and on the latter would risk fostering resentment and creating a legitimacy deficit in a climate where recent Foundation decisions – such as the non-implementation of the autoconfirmation trial, the roll-out of the Article Feedback Tool and the proposal for an image filter – have met with a difficult reception. Nor can the impact on under-prepared students and academics who were brave enough to implement a challenging and untested metric of assessment be ignored. The Indian experience offers much food for thought for future efforts. In the meantime, it may do well to bear in mind the maxim of Samuel Beckett: “Ever tried. Ever failed. No matter. Try again. Fail again. Fail better.”

This special report was made possible by reader research and the co-operation of Foundation staff; if you have an idea for a story or opinion essay, consider informing Signpost editors on the suggestions page, at the opinion desk, or by email at wikipediasignpost@gmail.com.


















Wikipedia:Wikipedia Signpost/2011-11-07/Special_report