JISC

This section provides profiles of the collections, including their condition, duplication with other collections, and factors influencing their selection and scheduling for scanning. It draws on survey work, visits and correspondence with those responsible for the collections.

• Collection surveys describes two collection-based surveys and their findings
• Collection profiles presents brief profiles of the individual collections, drawing on the surveys, visits and additional information
• Recommended strategies recommends strategies for addressing selection, copyright, de-duplication, and the scheduling of the collections, based on information in the preceding sections.

See the project workflow section to see how these activities fit within the overall project work plan and workflow.

Collection surveys

Survey of format and condition of pamphlets

The format and condition of the pamphlets will have an impact on: their selection (e.g. item may be too fragile to send); handling considerations; time required to scan (e.g. number of pages, tightness of binding); and the quality of the OCR output achieved (e.g. use of unusual typefaces).

In preparation for the initial project proposal, contributing libraries were asked to provide information about the size and condition of their collections. Libraries supplied a total number of pamphlets for their collections. Total page numbers were then estimated by multiplying the number of pamphlets by an average of 25 pages per pamphlet (for 3 collections) or 35 pages per pamphlet (4 collections, who said their average would be higher than 25). Based on these calculations, the initial project bid estimated that 1 million pages (the BOPCRIS capacity for this project) would approximate 30,000 pamphlets.

As part of the scoping work for the revised project proposal, a survey was undertaken to collect more accurate page statistics and determine other factors likely to affect scan time and image or OCR quality.

Methodology

Libraries were asked to check a random selection of 100 items drawn from the collections chosen for inclusion in this project. In five cases a computer program was used to generate this selection from lists supplied by the libraries; in two cases (Durham and UCL) the sampling was made on a systematic basis by library staff, but without sight of the pamphlet or its title.

Each library was asked to check a range of characteristics, such as size, location, condition of binding, and presence of greyscale or colour illustrations. These criteria were drawn up by Julian Ball (BOPCRIS Manager) and the author. They were based on experience and some initial testing in Southampton (see Digital datasets).

The following criteria were chosen for the format and condition survey:

A copy of a page from the survey form is included as Appendix A. Questions 1-18 on this form relate to this survey and follow the order of the criteria listed above. As the survey began it became clear that some collections included significant proportions of non-English language pamphlets, so libraries were additionally asked to gather this information. This was not always possible where the survey work was already underway.

This survey was undertaken during 4-11 August, with some libraries spending up to 10 hours on it. This time was provided as a partner contribution to the scoping study.

Results

The tables below provide findings from the format and condition survey. General discussion is provided in this section, with the significant characteristics of each collection detailed in Collection profiles below.

Table 2.1.1:A Location and page averages (based on answers to questions 2 and 9)

Note 1. In some cases a number less than 100 were checked, due to missing items or time constraints.
Note 2. Durham's sample included some 20th century items and other forms of publication, which may account for its higher page average.

Table 2.1.1:B Grey and colour pages (based on answers to questions 9, 10 and 11)

Note 1. Durham's sample included some 20th century items and other forms of publication, which may account for the higher number of grey pages.

Table 2.1.1:C Binding condition (based on answers to questions 3-6)

Table 2.1.1:D Pamphlets with foldings, annotations and adverts (based on answers to questions 12-15)

Table 2.1.1:E Pamphlets with unusual typefaces, multiple copies and foreign languages (based on answers to questions 16-17 and an additional question1)

Note 1. The foreign language check was introduced after the survey had begun when it appeared this would be significant for some collections. Consequently, not all libraries recorded this information.

Table 2.1.1:F Size, loose or missing pages and overall suitability for scanning (based on questions 1, 7-8, 18)

Note 1. Durham's sample included some 20th century items and other forms of publication, which may account for the larger sizes.

Discussion

Some care needs to be taken in drawing conclusions based on this survey, since only 100 (or less) items were assessed from each collection. For the smaller collections (Durham and Liverpool) this represents 6 or 7% of the collection; but for the larger collections (LSE and Bristol) it is less than 1%. Nonetheless, it provides better information than the previous estimates and enables some profiling to be done. Should the project go ahead it would enable some of these statistics to be checked and would provide an indication of how accurate and useful this approach to characterising collections is.

Collections profiles provides a brief profile of each collection and highlights characteristics identified by the condition survey that are likely to impact upon scan time. These and other factors noted in that section have influenced the scheduling of the collections (both their order and allocation of time).

One of the most important findings from the survey was that the page averages for these collections was much higher than the 25-35 previously estimated. Because the project had set the number of pages to capture at 1 million (the BOPCRIS capacity for this project), this meant reducing the overall number of pamphlets we would expect to capture. The revised bid recalculated each collection's page numbers based on the averages found in this survey (see Table 2.2.1:A above for averages). When combined with a reduction for anticipated duplication (see next section), this reduced the overall number of pamphlets from nearly 30,000 to just over 23,000. Table 2.1.2:B, below, presents new estimates for each collection. Note that because selections are to be made from the LSE and Bristol the numbers of pamphlets from these two collections were adjusted until 1 million pages was achieved.

Survey of duplication across collections

Published pamphlets are not unique items and with seven contributing libraries some level of duplication is to be expected. It is important for the project to gauge the extent and pattern of that duplication in order to determine: (a) the best place within the workflow to de-duplicate, (b) the best order in which to scan the collections, and (c) the likely reduction in number of scanned pamphlets due to de-duplication. Estimates of duplication from those familiar with this material ranged from 20%-60% (across Copac as a whole), so a more accurate measure was required.

Methodology

It was hoped that duplication could be gauged by an automated means using the CURL or Copac databases. This did not prove possible due to the existence of multiple records and the lack of time and resources to develop suitable tools for comparison. A part of the work of MIMAS in Work Packages 3 and 8 of the project (see 4.3 below) will be to develop such tools to enable all libraries holding a pamphlet that has been digitised to be provided with links. In the absence of an automated means, the same 100 item sample used for the condition survey (Collection surveys above) was checked by libraries against Copac for duplicates: (a) across the six other libraries contributing to Phase 1 and (b) against all holdings on Copac. This check was included as questions 19 and 20 on the survey form (see Appendix A).

The duplication survey took place alongside the format and condition survey, during 4-11 August. In some libraries the same staff members completed both surveys; in others, special collections staff undertook the format and condition survey (questions 1-18) and cataloguing staff undertook the duplication survey (questions 19-20). Checking 100 records on Copac took up to 6 hours for some libraries. This time was provided as a partner contribution to the scoping study.

Results

The table below presents findings from the duplication survey. General discussion is provided in this section, with comments relating to individual collections discussed in Collection profiles below.

Table 2.1.2:A Duplication survey results (based on questions 19 and 20 of survey form)

Note 1. There has been a delay in loading some of UCL's records into the CURL/Copac databases, so the duplication with its collection is not fully represented here. UCL's records will be loaded before the project commences.

Discussion

As previously mentioned, care needs to be taken in drawing conclusions based on this survey, since only 100 (or less) items were assessed from each collection. Nonetheless, it provides better information than previous guesses and enables some estimation to be done. Should the project go ahead it would enable the accuracy and usefulness of this approach to be evaluated.

Between 56 and 77 percent of these collections were found to be duplicated within at least one of the other 23 libraries represented on Copac. However, this still leaves a significant amount of material that is unique to each of these collections (23-44%).

Between 21 and 45 percent of the pamphlets in these collections were found to be duplicated within at least one of the other 6 libraries taking part in Phase 1 of the 19th Century Online Pamphlets project. However it is important to note here that the collections being digitised are a subset of each library's 19th century pamphlet holdings. For example, Manchester's Foreign and Commonwealth collection only represents about 21% of its total estimated holdings of 19th century pamphlets on Copac (17,000). So the 10% of UCL's Hume Collection found duplicated with Manchester's holdings is unlikely to equate to a 10% duplication for this particular project.

There is a fairly high level of duplication with the LSE and Bristol, which have large 19th century pamphlet collections (an estimated 15,989 and 22,150, respectively, on Copac). However, the project intends to make a selection from these two collections rather than capture them in their entirety, so the overlap with these collections can be compensated for in the selection process.

It is important to note, however, that the records for UCL's 19th century pamphlets are not yet fully loaded onto the CURL and Copac databases. It is likely that there is higher duplication with this collection than is apparent in the survey. UCL expect to have their full records on Copac before the project start date (January 2008).

For the purposes of putting together the revised bid we have assumed a duplication of half of the total amount found 'duplicated within any partner collection' for the Durham, Liverpool, Manchester, Newcastle and UCL collections and reduced the number of pamphlets we expect to capture accordingly. It may be that this reduction is larger than is necessary and we would seek to refine these estimates as the project proceeds. As selections are being made from Bristol and LSE, their numbers were not reduced for duplication, but were adjusted upwards to compensate for the reduction across the other collections, in order to make the collection up to 1 million pages.

The table below shows the combined effects of the higher page averages and the reductions made to account for duplication. Although it is hoped that this is closer than the previous estimates, it is likely that further adjustments will need to be made as the project proceeds and the numbers of pamphlets may rise or fall.

Table 2.1.2:B Effects of higher page averages and reduction for duplication

Collection profiles

This section provides profiles for each of the seven collections, including overviews of the collection content and a list of significant points that emerged from the surveys, or from visits and discussions with collection contacts. All collections were visited during the scoping study.

Preferences for transportation, insurance, storage, scheduling of collections, and other issues, were discussed with collection managers during the visits or by correspondence. In addition to specific issues included in the profiles below, several points can be made here that apply to all or most of the collections:

• Collections were presented with the option of doing their own scanning for this project, but all are happy to use BOPCRIS as a consortia scanning service
• Everyone was happy with the suggested transport arrangements (HarrowGreen, Momart or equivalent) and most preferred the transport company to pack the pamphlets
• Everyone was happy with the level of insurance cover provided by the transport firms we asked to quote (minimum of £100,000 per consignment) and by BOPCRIS whilst on premises (minimum of £100,000)
• Everyone was happy with the storage conditions being offered by BOPCRIS, which is temperature controlled, has low ultraviolet levels and is only accessible by authorised personnel
• No one said they would require the return of original items from BOPCRIS if requested by users whilst away: each would notify users of their collection's absence and be happy to make do with photocopies or digital images (if already scanned) from BOPCRIS
• Collections will have the option to receive digital datasets relating to their own items (images, metadata and OCR), but few intend to take up this option within the life of the project (Durham, Manchester and possibly UCL)

The key points identified for these collections have influenced the proposed scheduling.

Recommended strategies

During the scoping study we focused on four issues of particular importance in ensuring a good selection and flow of pamphlets to BOPCRIS for scanning: (1) selection of material; (2) addressing any copyright issues; (3) addressing the duplication across the collections; and (4) the scheduling of the collections for scanning. This section presents strategies for dealing with each of these issues. These would be firmed up in the project plan and further amended as necessary once the project was underway.

Selection strategy

As stated in the outline, an aim of this project is to provide users with a wide selection of pamphlets that focus on the political, social and economic issues of the 19th century. Instead of just digitising 23,000 pamphlets relating to this theme (which could have been achieved through a combination of two of the larger collections), this project has chosen to focus on capturing as much as is possible of a number of smaller collections associated with individuals or families (Durham, Liverpool, Newcastle and UCL) or organisations (Manchester), and to supplement these with pamphlets drawn from larger collections (Bristol and LSE). There are several advantages of capturing whole collections:

• In addition to understanding the motivation of pamphlet producers , users will be able to appreciate the motivations of their collectors . As seen in the format and condition survey above, some of these collections include significant levels of annotation by their collectors.
• Although these collections have a political, economic or social focus, they include material relating to the wider interests of their collectors, providing a good overview of the whole pamphlet literature.
• The collections can be linked with the collection-level descriptions in the online Guide to 19th Century Pamphlets; and found in collection-based searches.
• By including a larger number of library collections within this project, the workload is spread, the benefits are shared more widely (e.g. digital copies for libraries), and more can be learned to support the extension of the project to other collections in later phases.

De-selection criteria

For five collections, then, the appropriate strategy is not a selection strategy, but a de-selection strategy. Although the goal is to capture as much as is possible of these collections, the project anticipates de-selecting some pamphlets from digital capture, for these reasons:

• There are copyright concerns (see Copyright strategy)
• The pamphlet was published outside the bounds of the 19th century (either earlier or later). We note that extending into the 20th century would greatly increase copyright issues.
• The pamphlet has already been captured or is better captured from another collection (see De-duplication strategy)
• The pamphlet is too fragile to send for scanning.

Note that the project will not be deselecting on the basis of difficulties in scanning or OCR. BOPRIS has specialist scanners that can capture any material a library is willing to send.

Selection criteria

For the remaining two collections (Bristol and LSE), a positive selection of pamphlets will be made and will be based on the following criteria:

• Their relevance to themes of the great 19th century debates (e.g. universal suffrage, relationship of church and state, colonial policy), as identified by the collection curators, and the academics and teachers involved in the management and steering groups.
• Their usefulness in addressing gaps in the digital collection (e.g. themes not well covered, formats not represented, particular authors who should have a voice). Again, these gaps will be identified by curators, researchers and teachers.
• Feedback and demand from collection users : the bulk of Bristol and LSE material will be selected later in the project, by which stage there will be material available online and the possibility of tracking usage and surveying users.
• Replacements for copies held in the smaller collections that are in too poor or fragile a state to scan.

Copyright strategy

Copyright is the primary IPR issue facing a 19th century collection of published pamphlets. Although these collections are owned by their libraries (or have been included in the project with owners' permission), this ownership or permission does not automatically give libraries a right to copy them. For most of the pamphlets copyright will have expired and there will be no barrier to copying. But it is anticipated that a small proportion of the late 19th century material within these collections will still be within copyright and require either permission or exclusion. Project partners have some experience in these issues, but the project will seek advice during Work Packages 1 and 2.

We expect to identify three classes of material:

Class 1. Items clearly out of copyright (based on their age or anonymity)

Class 2. Items potentially within copyright due to age but of unknown status (e.g. because the death dates of the author or identity of their inheritors are not easily discoverable)

Class 3. Items known to be in copyright (based on age or the known dates of the author)

The following approach may be adopted:

• Libraries can send any Class 1 materials without concern
• Libraries must make the decision whether to send any Class 2 materials and bear the full responsibility for this decision: the project will expect them to make efforts to determine its status and require them to provide a suitable indemnification (this will form part of a Memorandum or Understanding)
• Libraries must not send any Class 3 materials for scanning without obtaining appropriate copyright clearance. The project will provide some assistance to libraries, including a suitable licence.

The following diagram shows a possible workflow for determining a pamphlet's copyright status.

Figure 1 Suggested Copyright Workflow

*This is a conservative cut-off - the project will seek legal advice on this.

Note that this chart assumes a conservative cut-off publication date of 1857 (150yrs) in case of an uncertain death date. We are aware of other projects who are using 120 years (i.e. 1887) as their cut off and will seek advice on this during the initial stages of project planning. It is anticipated that the vast majority of pamphlets in this collection will belong to Class 1. Many of the pamphlet records checked on Copac include dates for their authors, which will assist in determining the pamphlet's copyright status.

If, despite this strategy, issues arise with any of the pamphlets, the project would seek to remove it from the digital collection.

De-duplication strategy

As the duplication survey above has indicated, there is likely to be some duplication across these collections, although it is difficult to gauge the exact extent.

Because the project is selecting from the two larger collections (Bristol and LSE) there is an opportunity to de-duplicate these at the selection stage - provided there is some means of catching duplication when the selection is being made.

There are several approaches that might be taken to de-duplication for the smaller collections:

• Do not de-duplicate This may make sense given the focus on individual collections and the level of annotations of some pamphlets. However, this must also be balanced against the project's aim to create as large and accessible a selection of pamphlets as possible. Where there are a handful of personally annotated copies in different collections, the occasional duplicate may be permitted. But as a general policy, the approach of ignoring duplication would lead to wastage and reduce the number of unique resources that could be captured and delivered.
• De-duplicate at library, before sending for scanning Advantages of de-duplicating at the library are that pamphlets are not handled, transported, and removed from circulation unnecessarily. Disadvantages in de-duplicating at the library are that a comparison between different copies will not be possible, and the library going first in the queue will have more of their collection scanned than those following. There needs to be some mechanism for the LSE and Bristol to select around the other collections, so a database tool developed for this purpose could also be used to manage de-duplication between the smaller collections.
• De-duplicate at BOPCRIS, before scanning This approach has the advantage of enabling a comparison between physical copies. However, this would require that a pamphlet is kept until its duplicate arrives, which will take up space, delay return (and possibly require additional transportation). It would also require some means of identifying that the pamphlet has a duplicate in another collection.
• De-duplicate at BOPCRIS, post-scanning This approach was suggested in the initial bid, before the extent of duplication was explored. Although this does not require the originals to be held at BOPCRIS, it does assume that the appropriate datasets can be easily retrieved and compared. This may not be the case if BOPCRIS is sending the data to JSTOR and then deleting it from its own storage system (BOPCRIS does not have the resources to hold the entire digital collection at once). This approach also leads to great wastage, since one of the duplicate pair has been handled, transported and digitised unnecessarily.

There is no perfect solution to de-duplication. However, as a result of this scoping study the recommended strategy is to de-duplicate at the library, prior to sending the item to Southampton (Option 2). Library partners have agreed to this approach. It will require some sort of system (e.g. database) to identify duplicates among the smaller collections and enable the LSE and Bristol to make their selections with the other collections in view.

System to manage de-duplication

The project proposes that a database be developed at the beginning of the project ( see WP3) and maintained by a Project Officer located at BOPCRIS. We would expect to populate this database with simple bibliographic records exported from library OPACS or CURL/Copac at the beginning of the project. The database records would include additional fields to, for example, log the status and condition of the pamphlets.

As a library prepares its collection (or selection) it searches the database and identifies any duplicate records in other collections. It then checks the status of these items.

A pamphlet record can have one of several statuses:

1. Not checked - the default; for items that have not yet been checked or selected from a collection
2. Missing - pamphlet could not be found
3. Not suitable - pamphlet is too fragile to send
4. Sent - pamphlet is waiting to be sent or in transit
5. Received - pamphlet is at BOPCRIS and awaiting scanning
6. Scanned - pamphlet has been scanned and is awaiting signoff and return
7. Returned - pamphlet has been returned
8. Duplicate - not sent

If the librarian finds a duplicate with a status of 1-3, then the item should be sent (unless that library's own copy is also missing, not suitable or in poor condition). If another copy has a status of 5-8, then the item should be returned to its place on the shelf (or ignored within the volume) and the appropriate status recorded in the database.

Some libraries may still wish to check their duplicate for any significant annotations and make a case with the project team for a duplication.

The following diagram shows a de-duplication workflow based on the suggestions above. Note that it makes reference to a record slip. It is recommended that such a slip accompany all pamphlets to be scanned, and include details of the library name and catalogue ID number and other information. This will be particularly useful in marking which parts of a bound volume are to be scanned. The project would supply a record slip template and recommend that it be printed on an acid-free or PH-neutral paper.

The diagram below shows how de-duplication might be achieved within the library workflow using the proposed database.

Figure 2 Suggested De-duplication Workflow

Strategy for scheduling collections

The Gannt chart below sets out a possible scheduling of collections for digital production at BOPCRIS (see WP5). The time allocations take into account the size of the collections (i.e. number of pages) with some adjustment for its condition, based on the profiles in Collection profiles. The ordering of the collections is also influenced by the key characteristics identified in Collection profiles. We have tried to avoid the preparation of material overlapping or there being more than three libraries' pamphlets at BOPCRIS at any one time.

We have indicated single consignments for the smaller collections and two consignments each for the larger collections, although some of the smaller collections may also be broken in half or sent as two van loads. The collections we are doing in their entirety are scheduled earlier, and those from which selections are made (LSE and Bristol) generally occur towards the end of the schedule. This will enable the selections to be more informed and to minimise duplication. It will also enable the project to adjust the numbers from LSE and Bristol to ensure that the 1 million page target is met.

It must be stressed that this is one possible scheduling for the collections. Should the project go ahead the timetable would be negotiated with library partners.

Figure 3 A scanning schedule

See the scanning schedule