IIIF datasets in Europeana: A Scholar’s delight

Image Removed

The recently launched Europeana Media player brings Europeana into a new era of International Image Interoperability Framework (IIIF) compatible, interoperable and unified playout of audiovisual heritage material online. 

IIIF & Europeana Working Group

Image Removed

This dedicated IIIF & Europeana Working Group follows the first of the proposals from the Task Force 'Preparing Europeana for IIIF involvement.'

Impact Assessment report: EuropeanaTech and IIIF

Image Removed

IIIF datasets in Europeana: A Scholar’s delight

Image Added

The recently launched Europeana Media player brings Europeana into a new era of International Image Interoperability Framework (IIIF) compatible, interoperable and unified playout of audiovisual heritage material online. 

IIIF & Europeana Working Group

Image Added

This dedicated IIIF & Europeana Working Group follows the first of the proposals from the Task Force 'Preparing Europeana for IIIF involvement.'

Impact Assessment report: EuropeanaTech and IIIF

Image Added

This assessment looked at the impact of EuropeanaTech’s members, steering group and the Europeana Initiative’s work on IIIF - read a summary of the research and download the full report. 

https://iiif.europeana.eu/presentation/[RECORD_ID]/manifest
Accept: [ACCEPT]

https://iiif.europeana.eu/presentation/[RECORD_ID]/annopage/[PAGE_ID]
Accept: [ACCEPT]

https://iiif.europeana.eu/presentation/[RECORD_ID]/anno/[ANNO_ID]

Response v2.1:

Fulltext Resource

The full-text resource represents the (image or audio) transcription of a single content resource (e.g. a page of a newspaper or manuscript). A full-text resource can be accessed separately from an annotation or annotation page using the following method.

Request

https://api.europeana.eu/fulltext/[RECORD_ID]/[FULLTEXT_ID]

Parameter

Description

RECORD_ID

The identifier of the record which is composed of the dataset identifier plus a local identifier within the dataset in the form of "/DATASET_ID/LOCAL_ID", for more detail see Europeana ID.

FULLTEXT_ID

The identifier of the full text resource.

Response

The response is a JSON-LD structure composed of the following fields:

Parameter

Datatype

Description

@context

String (URL)

The URL of the JSON-LD context (always with the value "https://www.europeana.eu/schemas/context/edm.jsonld").

id

String (URI)

The canonical identifier of the full-text resource.

type

String

The type of the resource. Always set to "FullTextResource".

language

String

The more predominante language of the transcription text represented as a ISO 639 language code. Parts of the text may be written in different languages. When that is the case, the language information will be indicated as part of the full-text Annotations.

value

String

The transcription text.

Example: Requesting a full-text resource.

Request:

https://www.europeana.eu/api/fulltext/9200396/BibliographicResource_3000118435063/8ebb67ccf9f8a1dcc2ea119c60954111

{
  "@context": "https://www.europeana.eu/schemas/context/edm.jsonld",
  "id": "http://data.europeana.eu/fulltext/9200396/BibliographicResource_3000118435063/8ebb67ccf9f8a1dcc2ea119c60954111"
  "type": "FullTextResource",
  "language": "nl",
  "value": "… De ondergeteekende sedert veie jaren drukker van het met Uit. Oetober 11. vervallene Nieuw A. H. _ E. Blad, heeft de eer te berigten, dat hij, bewogen met het lot van eene menigte huisge zinnen die daardoor plotseling bij den naderenden winter hun be staan hebben verloren, besloten heeft tot de uitgaaf eener nieuwe courant onder de benaming van: Het Amsterdamsche Handels- en Effectenblad. en dat hij daartoe de voorloopige medewerking heeft verkregen van belangstellenden, die van oordeel zijn, dat het bestaan van een dagblad als het gewezen Nieuw A. H. _- E. 81. voor het algemeen … "
}

Searching on full-text

The full-text can also be searched using a separate Search API while the Newspapers Thematic Collection is in MVP. It supports the same functionality as the main API but under the following endpoint and with the addition of 2 search fields and 1 profile as described below. For more information on the other methods, see the Search API documentation.

Request

https://newspapers.eanadev.org/api/v2/search.json

Fields

Datatype

Description

fulltext

Text

Allows searching on the transcribed text (ie. full-text) of the item.

issued

Date

A true date field reflecting the date of the Newspapers Issue.

Profile

Description

hits

Displays the mentions in the transcribed text where the search keyword was found.

Parameter

Datatype

Description

hit.fl (optional)

List (String)

A comma- or space-separated list of fields from which hit highlighting should be generated. A wildcard of “*” (asterisk) can be used to match multiple fields, such as “fulltext.*” or even “*” to highlight on all fields where highlighting is possible. If omitted default to “*”.

hit.selectors (optional)

Number

Specifies the maximum number of highlighted selectors (ie. snippets in Solr) to generate per result (ie. record). If omitted defaults to 1. It is possible for any number of selectors from 1 to this value to be generated, up to a limit of 10.

Example: Searching on full-text and showing hit highlighting.

Request:

https://newspapers.eanadev.org/api/v2/search.json?query=paris&profile=hits&wskey=APIKEY

{
  "apikey": "api2demo",
  "success": true,
  "requestNumber": 999,
  "itemsCount": 12,
  "totalResults": 439085,
  "items": [ ... ],
  "hits": [
    {
      "scope": "/9200303/BibliographicResource_3000059897585",
      "selectors": [
        {
          "field": "rdf:value",
          "exact": "pāri",
          "prefix": "Vi sai mūsu lielajai valstij partijas XVIII kon ferencei par godu gājis ",
          "suffix": " varens sociālis tiskās sacensības vilnis. "
        }
      ]
    },
    ...
  ]
}

Accessing images in high resolution: downloading data

To foster the reuse of the data that is published in Europeana as part of the Newspapers Thematic Collections, we make both the metadata and the full-text available for bulk download as compressed zip files. The metadata is available as CC0 the same way as all the metadata exposed via the API (see Terms of Use) while the full-text is available as Public Domain Mark.

List of datasets

The table below lists all the datasets that are published and available for download. If you are looking for the complete text of a Newspaper then we suggest using the (4) option, as opposed to using (3) where the trascription is partioned per page.

Given the fact that the files are very big and can take many hours to download, as an alternative to download directly via the browser, you can login to the FTP server at "download.europeana.eu" with username "anonymous". This will allow you to resume if the download gets stuck.

dataset number

Metadata¹

Full-text (ALTO)²

Page level full-text (EDM)³

Issue level full-text (EDM)⁴

9200300

download

 (229M) (MD5)

download

 (63G) (MD5)

download

 (116G) (MD5)

download

 (113G) (MD5)

9200301

download

 (37M) (MD5)

download

 (13G) (MD5)

download

 (20G) (MD5)

download

 (20G) (MD5)

9200338

download

 (213M) (MD5)

download

 (158G) (MD5)

download

 (278G) (MD5)

download

 (277G) (MD5)

9200339

download

 (39M) (MD5)

download

 (11G) (MD5)

download

 (21G) (MD5)

download

 (17G) (MD5)

9200355

download

 (212M) (MD5)

download

 (97G) (MD5)

download

 (159G) (MD5)

download

 (157G) (MD5)

9200356

download

 (137M) (MD5)

download

 (40G) (MD5)

download

 (17G) (MD5)

download

 (17G) (MD5)

9200357

download

 (23M) (MD5)

download

 (5G) (MD5)

download

 (9G) (MD5)

download

 (9G) (MD5)

9200396

download

 (4M) (MD5)

download

 (849M) (MD5)

download

 (2G) (MD5)

download

 (1G) (MD5)

Legend:

1. The original metadata in EDM XML format before being ingested into Europeana. There are slight differences between this data and the one published. For more information see the /wiki/spaces/EF/pages/2385313809.

2. The full-text encoded using ALTO (Analyzed Layout and Text Object) as it was delivered to Europeana. The ALTO is an open XML Schema meant to describe text coming from OCR and layout information of pages for digitized material. For more information see the official documentation page at the Library of Congress.

3. The full-text encoded using the EDM profile for IIIF fullltext after being preprocessed for publication in Europeana. A note that as opposed to the format used by the API (ie. JSON-LD), the data is in RDF/XML as it is the format used for ingestion into Europeana.

4. Very similar to (3) but wih the full-text represented at the Issue level. This means that the edm:FullTextResource will convey the complete transcription of the Newspaper.

Dataset structure

On each compressed zip file, there will typically be a file per each item (ie. metadata or issue level full-text) or page (ie. ALTO and page level full-text) with the following structure:

Item

DATASET_ID/LOCAL_ID.xml

Page

DATASET_ID/LOCAL_ID/PAGE_ID.xml

Retrieving images or portions of the image

The IIIF Image API specification offers the ability to request portions of the image in different resolutions allowing the image to be served progressively to a web client. Providers that have implemented the IIIF Image API for their content may indicate the availability of the image service using a dedicated profile. When this is the case, the Manifest will refer to the image service so that web clients can make use of it. Europeana also has available high resolution scans that were produced under the Europeana Newspapers project that are served using IIIF Image API.

Image service

Retrieves an image by applying the operations indicated in the path. The image endpoint used is from the Europeana Newspapers project.

Request

https://iiif.europeana.eu/image/[IMAGE_ID]/[REGION]/[SIZE]/[ROTAION]/[QUALITY].[FORMAT]

Image service details

Retrieves information about the image and the available operations that can be done on the image. The image endpoint used is from the Europeana Newspapers project.

Request

https://iiif.europeana.eu/image/[IMAGE_ID]/info.json

Retrieving full-text content

The term full-text is meant to refer to the correlation between the content resource (e.g. image, audio or video) and its textual representation (ie. transcription, subtitle, caption). In the EDM profile, the textual representation of the content resource is referred to as Full-Text Resource while the relations between the segments of the text and the coordinates in the image are referred to as Annotations.

Annotation Pages

An Annotation Page contains all the annotations that make up the full-text of a content resource (ie. image, audio or video). It is referred to by the Manifest and can be accessed via the following request.

Request

https://iiif.europeana.eu/presentation/[RECORD_ID]/annopage/[PAGE_ID]
Accept: [ACCEPT]

Response v3:

Response v2.1:

Annotation

An Annotation specifies a single relation between the full-text resource and the content resource (ie. image, audio or video). It is referred to by the Annotation Page and can be accessed via the following request.

Request

https://iiif.europeana.eu/presentation/[RECORD_ID]/anno/[ANNO_ID]

Response v3

Response v2.1

Fulltext Resource

The full-text resource represents the (image or audio) transcription of a single content resource (e.g. a page of a newspaper or manuscript). A full-text resource can be accessed separately from an annotation or annotation page using the following method.

Request

https://api.europeana.eu/fulltext/[RECORD_ID]/[FULLTEXT_ID]

Response v3

Response v2.1 (not supported)

Searching on full-text

There are two methods for searching on full-text, one for searching across all items where fulltext is available and a second to search within the fulltext of a single item.

Search accross all items with full-text

This method adopts the same API structure and functionality as the Search API with the addition of two search fields and one profile as described below. For more information on the other methods, see the Search API documentation.

Request

https://api.europeana.eu/fulltext/search.json

Search within full-text of a single item

For a Newspapers item, this will mean searching in the text of all the pages that make up the Newspaper.

Request

https://iiif.europeana.eu/presentation/[RECORD_ID]/search

Response v3

Response v2.1

Source code and changelog

As mentioned in the introduction, the IIIF APIs are made up of several distinct APIs, each one with its own source code project in GitHub and changelog as listed below.