1 Goal of the Metis Sandbox

The Metis Sandbox is a test environment for your data. It consists of a set of tools with which you can:

1. simulate ingesting and running the Metis workflow on your data,

2. see what your records would look like on the actual Europeana.eu portal,

3. get insight into the quality of your records.

2 Where to find the sandbox

The Sandbox can be accessed through https://metis-sandbox.europeana.eu/ . 

3 How to prepare your dataset

A dataset for the Metix Sandbox can currently not exceed 1,000 records. If your dataset is larger than that, you’ll see a warning message indicating that only the first 1,000 records will be processed. A dataset should contain one record at minimum. Your dataset needs to meet the requirements of the Europeana Data Model (EDM) external.

Your dataset can be uploaded as a zip file or it can be sent via HTTP (i.e. zip file download) or OAI protocols. Optionally an XSLT file can be supplied with the dataset.  If this is provided, the Metis Sandbox Server will try to transform the dataset into the correct format after it uploads it and before validating it against the EDM external specifications.

4 General interface elements

There are several different methods to interact with the sandbox. Below is a list of the general interface elements and their uses.

4.1 Step indicators

The step indicators show your location within the Sandbox. They behave as tab headers: clicking on a step will navigate to the corresponding section, with the active step's name appearing above.  A step indicator can correspond to the "upload", "track" or the "record report" step.

Each step indicator will change shape, colour and border as you fill out the form.  The active step indicator's border will have a brighter coloured, and the indicator will turn square-shaped and orange-coloured when your data is valid and can be submitted.

In addition, a step indicator can display a step's state.  Here the tick under the step indicator shows that the step has been performed…

...whereas here the cog indicates that the step is busy.

4.2 Submit, Track and Report Buttons

The submit, track and report buttons allow you to upload a dataset, track its processing and inspect its records when the processing is complete.  These buttons are grey (disabled) when required information is missing in the corresponding step’s form.  The buttons become green (and clickable) when you enter correct information into the field(s), and the form can be submitted.

4.3 Internal Links

Some sections of the Metis Sandbox contain links that when clicked will take you to a different step, and will enable that step if it is not already enabled. The internal links are found at the bottom of the UI.  For example you can enable (and navigate to) the “Upload Dataset” step by clicking this link at the bottom of the “Track Dataset Processing” step:

From the “Record Report” step you can navigate to either the “Upload Dataset” or the “Track Dataset Processing” step:

5 The Home screen

The default view, the screen you land on when navigating to the tool, allows you to track an existing dataset, request information about a record within that dataset or create a new dataset. It looks like this:

A. Step Indicator: indicates that "Track Dataset Processing" is the current step.  Once other steps become available then clicking this will return you to this step.
B. Dataset Id Input: used to enter the id of a previously uploaded dataset.
C. Record Id Input: used to enter the id of a record within the specified dataset. It enables when a dataset id is entered.
D. Track Button: submits the value in the dataset id input. It enables when a dataset id is entered and, when clicked, takes you to the “Track Dataset Processing” functionality (see below).
E. Track Button: submits the value in the record id input. It enables when a record id is entered and, when clicked, takes you to the “Record Report” functionality (see below).
F. Create New Dataset Link: enables and navigates to the “Upload a new Dataset” functionality (see below).

6 Upload a new dataset

To create a new dataset click on the “create a new dataset” link at the bottom of the home screen (F in the image above). This will append “/new” to the page url and bring the “Upload Dataset” step into view.

6.1 The Upload Form

The “Upload Dataset” view looks like this:

A. Step Indicator: indicates that "Upload Database" is the current step.  If you switch to another step then clicking this will return you to this step.
B. The dataset name input field.  A dataset name is valid if it contains only letters, digits and the underscore character (‘_’).
C. The dataset country drop-down.
D. The dataset language drop-down.
E. The harvest protocol radio button set.
F. The zip file input.  This appears because “file upload” is the selected protocol.  If the selected protocol is changed to “OAI-PMH upload” or “HTTP upload” then an alternative field (or set of fields) will appear here.
G. An (optional) checkbox to specify that you want the Metis Sandbox Server to transform your dataset using XSLT.  If selected then a file input will appear below it allowing you to upload an XSL file.
H. Step Indicator (inactive): clicking this will take you to the “Track Dataset Processing” step.
I. The “Submit” button: enables when all the (obligatory) fields have been completed.

A valid dataset name should be entered into the “name” field, and a selection made from the country and language drop-downs. If you want to upload a dataset in the zip format then use the zip file input to select the zip file from your local computer.  If on the other hand you want to perform an upload using the OAI-PMH or HTTP protocols then you should change the selected value of the harvest protocol radio button set.

6.2 The Harvest Protocol

6.2.1 Zip File

By default the harvest protocol is by zip file (containing your records):

6.2.2 OAI-PMH

If you set the harvest protocol to OAI-PMH then you should enter values for the harvest URL, the metadata format, and optionally a value for Setspec:

6.2.3 HTTP

If you set the harvest protocol to HTTP then you should enter a value for the Url, this should be the download location of a zip file that contains your records:

6.3 XSL Transformation (Optional)

If you check the checkbox labelled “Records are not provided in the EDM (external) format” then an additional file input will appear allowing you to specify an XSL file:

6.4 The Dataset ID

Once all the fields have been filled the “Submit” button will become enabled. Click it to trigger the processing of your data. The Metis Sandbox will redirect you to the “Track Dataset Processing” step where you will see your data being processed in real-time.

In this window you will also be able to find your dataset ID (for details on how to do that, see the relevant section on the “Track Dataset Processing” functionality below). Each uploaded dataset gets a unique ID. You will need to remember or save this ID for yourself to be able to get back to your dataset later (for instance from the home page, see above).

7 Track dataset processing

To track (monitor) the processing of a dataset that you uploaded earlier, or see the results after it finishes processing, enter your dataset ID in the home screen and click the ‘track’ button (see above).

7.1 The Data Processing View

A submitted dataset id will bring up the track dataset processing view. It will also change the page’s url to reflect the id of the dataset processing being displayed.  The dataset processing view looks like this:

A. The dataset name.  The tick after the dataset name indicates that processing is complete
B. An (optional) flag indicating whether the dataset was xsl-transformed
C. The processing date, preceded by an (optional) flag indicating that not all records in the dataset were processed
D. The country and language of the dataset
E. The processing steps performed on the dataset (they correspond to the list of items just below, element F).
F. The details of the processing steps performed on the dataset
G. The (optional) warning indicating that not all records in the dataset were processed
H. The (not enabled) record id field
I. The dataset ID of the current dataset
J. A link to the dataset preview as it would look like on Europeana

To track the data processing of a different dataset just replace the value in the dataset id field with another id and click the “track” button.

7.2 The Metis workflow

These are the steps the data will go through as part of the processing workflow:

1. Harvest (H): how many records of your total dataset have been imported successfully

2. Validate(edm external(Vi): how many records passed the validation of the external EDM

3. Transform (T): how many records have been been transformed from the external EDM format to the internal EDM one

4. Normalise (N): how many records have been normalised. Normalisation acts on individual values in the data and could include deletion of double spaces and of duplicate values.

5. Enrich (E): How many records have been successfully enriched with the information of processing the data.

6. Process media (M): for how many records the linked media could be found

7. Publish (Pu): how many records records were published in a copy the Europeana website (see chapter 7)

The colours of each step indicate how successful this step was:

• Green: step completed without errors, all records are considered for ingestion

• Yellow: non critical warning. Problems with the records have been detected, but the records will still be considered for ingestion.

• Red: critical warning. Incomplete records have been detected, these records will no longer be considered for ingestion.

7.3 The Data Processing Errors Window

Shown below is an example of a dataset that processed with many errors:

A. A link to the errors window
B. The bold font of the number indicates that this is another link to the errors window
C. No report is available for this error, so the the number does not have a bold font and there is no link to the errors window

Clicking a link to the errors window will open it, allowing you to see the error detail:

7.4 View the published records

Click on “view published records” (item J in the image in 7.1)  to view your final data in a copy of the Europeana website.

8 Record report

To view the details of a dataset record you have to enter both the id of the dataset as well as an id of the record you want to view in the home screen, and then click the ‘report’ button (see above).  

8.1 Record Provider Ids and Europeana Ids

Every processed record has both a Provider id and a Europeana id. 

• A Europeana id begins with a forward slash followed by the record’s dataset id, another forward slash and then a further sequence of (non-whitespace) characters. You can find the Europeana ID of a specific record by clicking the dataset preview link and finding and inspecting the records there.

• A record’s Provider id, on the other hand, can be any sequence of (non-whitespace) characters, and is the value that can be found in the ‘rdf:about’ attribute of the ‘providedCHO’ section of your record.

You can search for a record using either of these record ids, so the “Report” button will enable itself when any sequence of non-whitespace characters has been entered into the record id field.  If, however, the UI detects that you’ve entered an id that matches the format of a valid Europeana record id, then it will show a line connecting the record id with the dataset id, as shown here:

A. The record id begins with a slash followed by the dataset id, so the id fields are shown as connected
B. The “Report” button can now be clicked

8.2 The Record Report

The record report is divided into two main sections:

• the content tier section

• the metadata tier section

You navigate between these sections by clicking on the corresponding navigation orbs, highlighted as “C” in the diagram below.   The computed value of each section is shown below these navigation orbs.  In the illustration below these are “3” (for the content tier) and “A” (for the metadata tier).

A. Step Indicator: the inactive "Track Dataset Processing" step, if clicked, will return you to the dataset tracking step
B. The Record Report summary: top-level information about this record
C. Report Navigation Orbs: these allow you to switch between the "content tier" and "metadata tier" sections of the report
D. Content Tier Information: data about the record's content tier
E. Media Navigation Orbs: you can navigate multiple media items from here
F. Processing Errors: record processing error information appears here
G. Step Indicator: indicates that "Record Report" is the current step

8.3 Content Tier Media Information

The media information appears under the content tier breakdown section.  If there are 5 or fewer items, then a navigation orb corresponding to each item will appear.  The icon of each navigation orb illustrates the type of media item, as shown here:

If there are more than 5 media items available in the record report then the navigation orbs will be replaced with navigation arrows, an editable field and a spinner allowing you to browse the items or jump directly to a specific one, as shown here:

8.4 Metadata Tier Information

You can see the record report’s metadata tier information by clicking on the metadata tier navigation orb.  Metadata tier information is split into three sub-sections:

• Language dimension

• Enabling Elements Dimension 

• Contextual Classes Dimension 

These, like the main sections of the report, are navigable by clicking on the corresponding navigation orb.

Active language dimension

Active enabling elements dimension

Active contextual classes dimension

9 Troubleshooting

Dataset not found

Every two weeks the sandbox is emptied. It is highly possible that the dataset has been removed because of this.