evaluation


On Friday 19th September 2008 we signed-off the development phase of the OARS Project after the sixth and final development cycle. At the beginning of this cycle we agreed that the basic deliverables were:

  • Make it possible to re-ingest PDF, text, and thumbnails when editing metadata
  • Re-sort by relevance, title and date in search results, by clicking on a heading on this page
  • Enhance search engine with regard to foreign language text and search terms
  • Allow items to be ‘hidden’ so that they don’t show up in User Interface searches

The final functionality was delivered by Aptivate on Monday 8th September 2008. The only functionality that was not factored in by agreement was ‘re-sort by relevance, title and date in search results’.

During the 2 weeks that followed, the application and its data was loaded into its final locations for live running - i.e. all repository data and its search engine hosted at the Oxford University Research Archive (ORA), and the ‘front end’ application hosted here at Forced Migration Online (FMO). Once that was completed, final acceptance testing was conducted by the FMO team, which successfully led to the sign-off.

The remaining tasks are to customise the application to add help pages, create styling for all web pages, and to configure url addresses etc to comply with the existing FMO web site. This work is being undertaken by the FMO team and is due to finish at the end of November 2008, at which point the new OARS repository system will become a live component of the FMO web site.

On Wednesday 23rd July we had the fifth meeting with our developers, which happens at the end of each development cycle (typically lasting 2 weeks). At the beginning of this cycle we agreed that the basic deliverables were:

  • Add language as a filter on advanced search
  • Show ‘Document Type’, ‘No. of pages’ and ‘Sourced From’ (formerly ‘Collection’) on search results page
  • Show thumbnail, ‘No. of pages’ and ‘File Size’ on document / journal ‘landing page’ (i.e. metadata detail page), and facilitate ingestion of thumbnail along with document / journal
  • Add css classes to web pages to facilitate page desgin

The functionality was delivered on Wednesday 23rd July. There are some outstanding issues to be resolved for this development cycle, and these will be resolved in the sixth and final cycle. Once the development and initial bug fixes are complete for the final cycle we will move the data (i.e. Fedora Repository) and the user/management interface to their live locations before embarking on final acceptance testing.

The priorities and deliverables for the final cycle were agreed, and briefly are:

  • Make it possible to re-ingest PDF, text, and thumbnail when editing metadata
  • Re-sort by relevance, title and date in search results, by clicking on a heading on this page
  • Enhance search engine with regard to foreign language text and search terms
  • Allow items to be “hidden” so that they don’t show up in User Interface searches

On Wednesday 2nd July we had the fourth meeting with our developers, which happens at the end of each development cycle (typically lasting 2 weeks). At the beginning of this cycle we agreed that the basic deliverables were:

  • Ingest and index full text and all metadata from a new PDF document
  • Add metadata field search - Title, Author, Collection etc
  • Enable full index of all articles in journal issues for searching
  • Enable editing of all metadata items in Management Interface, and allow certain metadata items to have multiple entries

The functionality was delivered on Wednesday 2nd July. There are some outstanding issues to be resolved for this development cycle, and these will be resolved over the next cycles. This meeting was combined with user evaluation sessions devised to solicit feedback about the way in which the current repository interface is used and to refine the functionality required of the new interface.

There were 7 participants, ranging from novice to expert users of FMO. Each session enabled one participant to use the current interface and comment on its strengths and weaknesses. The feedback from these sessions was very interesting. As we supposed, some of the more esoteric functionality of the current system is not used/understood, and what seems to be required is pretty much what we are developing - a more open system, with greater searchability via search engines such as google, better access from subject portals, and a slightly simpler and more intuitive user interface. Several recommendations for improved functionality were expressed and will be factored into the ongoing development.

The next development cycle will be followed by a development meeting on Wednesday 23rd July. The priorities and deliverables for this next cycle were agreed, and briefly are:

  • Add language as a filter on advanced search
  • Show ‘Document Type’, ‘No. of pages’ and ‘Sourced From’ (formerly ‘Collection’) on search results page
  • Show thumbnail, ‘No. of pages’ and ‘File Size’ on document / journal ‘landing page’ (i.e. metadata detail page), and facilitate ingestion of thumbnail along with document / journal
  • Add css classes to web pages to facilitate page desgin

On Tuesday 17th June we had the third meeting with our developers, which happens at the end of each development cycle (typically lasting 2 weeks). At the beginning of this cycle we agreed that the basic deliverables were:

  • Index full text and metadata of stored items for search engines
  • Improved functionality with regard to search results lists
  • New functionality to allow view and download of full text for a document
  • New functionality for inserting journal issues into the repository and indexing their full text

The functionality was delivered on Monday 16th June. There are some outstanding issues to be resolved for this development cycle, and these will be resolved over the next cycles. The next development cycle will be followed by a development meeting on Wednesday 2nd July. The priorities and deliverables for this next cycle were agreed, and briefly are:

  • Ingest and index full text and all metadata from a new PDF document
  • Add metadata field search - Title, Author, Collection etc
  • Enable full index of all articles in journal issues for searching
  • Enable editing of all metadata items in Management Interface, and allow certain metadata items to have multiple entries

On Monday 2nd June we had our second meeting with our developers, which happens at the end of each development cycle (typically lasting 2 weeks). At the beginning of this cycle we agreed that the basic deliverables were:

  • Improved functionality around searching for documents, including use of search results lists
  • Improvements to the management interface regarding metadata add/edit
  • New functionality for inserting documents into the repository

The functionality was delivered on Friday 30th May and most was signed-off by the OARS team at the meeting. There are some outstanding issues to be resolved for the second development cycle, and this will be included in the next cycle of development, due to be completed on Monday 16th June. This will be followed by a development meeting on Tuesday 17th June. The priorities and deliverables for this next cycle were agreed, and briefly are:

  • Index full text and metadata of stored items for search engines
  • Improved functionality with regard to search results lists
  • New functionality to allow view and download of full text for a document
  • New functionality for inserting journal issues into the repository and indexing their full text

On Wednesday 14th May we had a meeting with our developers, which happens at the end of each development cycle (typically lasting 2 weeks). At the beginning of this cycle we agreed that basic functionality would be delivered around 2 scenarios covering the web-based user and management interfaces, and the underlying repository technology. These were:

  • Finding and printing a known document
  • Editing document metadata

The functionality was delivered on Tuesday 14th May and was signed-off by the OARS team at the meeting. The work done so far now provides a base for the next cycle of development, due to be completed on Friday 30th May, followed by a development meeting on Monday 2nd June. The priorities and deliverables for this next cycle were agreed, and briefly are:

  • Improved functionality around searching for documents, including use of search results lists
  • Improvements to the management interface regarding metadata add/edit
  • New functionality for inserting documents into the repository

On Monday 28th April the software development part of the OARS project started. On Monday and Tuesday 28/29 April we had ‘Kick-off’ meetings with developers from our consultants (Aptivate). The two days were extremely productive, during which we discussed the schedule for for deliverables and testing, access to computing facilities and access to the repository at Oxford University Research Archive.

Basically increasing functionality will be delivered to the OARS Team on roughly a 2-weekly cycle over the next 12 weeks. The OARS team will test the delivery, advise on functionality issues and problems found, then set the priorities / deliverables for the following 2 weeks, and so on.

There will be an evaluation of the development process itself, with a view to highlighting the challenges we face and hopefully to provide lessons for future projects. This is a very important part of this phase of the project and will be managed by our evaluation consultant Jan Rae.

On the 31 March 2008, the OARS project team hosted a meeting bringing together representatives of institutions from its network of advisors from the higher education (HE) sector in the UK.

The Forced Migration Online (FMO) team established a network of advisors for the OARS project in November 2007. It is comprised of researchers and academics from higher education (HE) institutions, as well as some students. We asked network members to complete a brief questionnaire about their usage, opinions and expectations of FMO. This information is being used to inform the team in its work to make the enhancements to FMO through the OARS project. The results of this user survey will be made available in the coming weeks.

At the meeting held in March in Oxford, discussions centred around how participants (as well as their colleagues and students) currently use FMO, and what they find particularly useful and not-so useful. This led onto a discussion about how FMO might be improved to better facilitate colleagues in their work. Among other things, it was agreed that the Web Catalogue could probably be scrapped as an independent tool and its contents largely be reallocated elsewhere on FMO; a catalogue of academic and training courses on FMO would be a useful tool; and we should collaborate in attempting to develop a process whereby copyright is sought to include in the FMO repository all relevant Masters and doctoral theses emanating from HE institutions in the UK. It was also agreed that staff or students from these institutions might contribute to FMO’s collection of research guides and resource summaries.

A summary of these discussions will be circulated among network members before being made public on the OARS website. The meeting held in March was held in parallel with a meeting of the OARS Steering Committee, where technical issues of the OARS project development were discussed at length.

In November 2007, the FMO team established a network of advisors for the OARS project, comprised of representatives of higher education (HE) institutions in the UK. Students, researchers and academics from universities and colleges comprise the majority of FMO’s users. It is therefore important to involve them in the FMO portal’s redevelopment and enhancement.

On behalf of the FMO team, the Director of the Refugee Studies Centre (RSC), Professor Roger Zetter, wrote to academic departments and centres in the UK requesting their input. We are hoping to conduct this consultation with as many university departments and centres in the UK as possible that conduct research, teaching and/or dissemination activities in issues related to migration, conflict, development and human rights. We are asking them to complete a brief questionnaire about their usage, opinions and expectations of FMO. This information will be used to inform the team in its work to make the enhancements to FMO through the OARS project.

In the interests of transparency and encouraging further information sharing and collaboration, the feedback received will be consolidated into a summarised report and shared with network respondents. We will also provide regular updates on the progress of the HE network’s input.

While for practical reasons this advisory group is formally comprised of representatives from HE institutions in the UK, the FMO team would strongly encourage all users of FMO - whether or not they be in the HE sector and/or in the UK - to provide us with feedback. In this way, they will be contributing to the OARS project by assisting the team in making the most appropriate changes to FMO to benefit all users.

The questionnaire on FMO usage can be completed and submitted online. Alternatively, the questionnaire can be downloaded as a text document and returned to the FMO team by email (fmo@qeh.ox.ac.uk), fax (01865 270721) or post (FMO Team, Refugee Studies Centre, Dept. of International Development, 3 Mansfield Road, Oxford OX1 3TB).