Goal of the Metis Sandbox

The Metis Sandbox is a test environment for your datasets. It consists of two tools: 

1. to get insight in your records and 

2. preview what your records will look like on Europeana.eu.

Where to find the sandbox

The Sandbox can be accessed through https://metis-sandbox.europeana.eu/ . 

How to prepare your dataset

A dataset for the Metix Sandbox can currently not exceed 1,000 records. If your dataset is bigger than you’ll see an error message. [ A future update will display a message that the first 1000 records will be used in the data archive  instead of showing an error.]

Your dataset needs to meet the requirements of the Europeana Data Model external.Your dataset should be on your computer or on a web address as a zip file. The zip file will be unzipped entirely after uploading.

General interface elements

There are several different methods to interact with the sandbox. Below is a list of the general interface elements and their uses.

Step indicators

The step indicators indicate the user’s location within the Sandbox. They behave as tab headers: clicking on a step will navigate to the corresponding section. 

Navigation buttons

Navigating between the steps can also be done by using the navigation buttons.

The Previous step button: allows you to go back one page

The Next step button: allows you to go to the next page

Track and submit buttons

The Track and Submit buttons allow you to track or submit a dataset. These buttons are grey (disabled) when required information is missing in the form.  The buttons become green (and clickable) when you enter correct information into the field(s) and the form can be submitted. 

Links

Navigation Buttons

	Background colour: green Border: light / green Shape: round	Default state: the step has not been completed yet (green background), the step is not active (light / green border) and the data is not submittable (round shape)
	Border: heavy / yellow	Active state: the step is active (heavy / yellow border) and the content for that step is being displayed
	Background colour: orange	Valid state: the form information for this step has been correctly specified (but is not yet submittable)
	Shape: square	Submittable state: the form information for this step has been correctly specified and the data is submittable
	Indicator: spinner	Submitted state: the form information for this step is being submitted
	Indicator: tick Background: green Shape: round	Processed state: the form information for this step has been submitted and processed (tick).  The step is not active (green border)
	Indicator: tick Background: orange Border: heavy / white Shape: square	Processed and submittable state: the form information for this step has been submitted and processed (tick).  The step contains valid information (orange background) that can be submitted (square shape).  The step is active (heavy border)

Home screen

The home screen gives you the option to track an existing dataset or create a new dataset. The options in the home screen are:

1. The menu toggle makes the different steps of the sandbox visible.

2. The Dataset ID bar can be used to enter the ID of a previous uploaded dataset

3. The Track button allows you to track the dataset entered in the dataset ID bar

4. The Create new dataset link can be used to start uploading a new dataset

Dataset ID

Each uploaded dataset gets a unique ID (see point 6.1 below). You will need to remember or save this ID for yourself to be able to get back to it later. Enter the dataset ID if you want to track its progress on different steps in the text field of the homepage and click on the “track” button, or to query the publish metis sandbox in a different  tab/window browser

Create a new dataset

Click on “create a new dataset” on the Homepage or the Dataset Processing page. This will take you to the Dataset Name page.

Enter a Dataset Name 

The dataset name is for you to identify the dataset yourself. A name does not have to be unique, because a unique ID is assigned to each dataset by the Sandbox. It is advised to choose a name that will make it easy for you to remember which dataset you’ve created and any details of that dataset. Spaces are not allowed in the name of the dataset. 

Enter a name in the text field and click on the “Next step” button.

Enter the Dataset Details

In the “Dataset details” screen you need to add the details of the dataset. You will need to indicate the country and the language of the dataset from a drop down menu. The drop down menu is searchable: click on the down arrow and type the first letters of the language or country to jump to the country or language in the list. Click on the “Next step” button to go to the next step.

Configure the Data Source

There are two options to upload your records to the sandbox [TBC: OAI-PMH upload is not enabled currently. This functionality will support harvesting of the records, comparable to current functionality in Metis.]:

1. File upload: upload a file from your computer

2. HTTP upload: Add an URL of the zip file with the records on a web server

This field is mandatory. Zip Files are unzipped by the Sandbox. [MacOS has the tendency to add files to zip files, these extra files are automatically skipped. ]

Upload a file 

1. Select the radio button left of “File upload”. A “Browse” button will appear.

2. Click on the “Browse” button. A pop up will appear to search for the file on your computer. 

3. Select the file you wish to use. Click on OK in the pop up window.  The Submit button will be enabled once a file has been selected.

4. Click the “Submit” button to start uploading the file.

The upload time depends on the size of the file and the speed of your internet connection. 

[TBC: February 2022 update: Additional developments are in progress, such as imposing a limit of 1 terabyte per upload.]

Use HTTP Upload

1. Select the radio button next to “HTTP upload”. A text bar will appear.

2. Paste the entire url of the zipfile in the text bar, including http://.

3. Click on the “Submit” button to start uploading the file.

Processing and analysing

Processing dataset

A tracking screen will appear after uploading. The tracking screen displays

1. The name of the dataset

2. The tracking number

3. The date and time when the dataset is submitted in the Sandbox

4. The details of the dataset (language and country)

5. The steps and results of the data ingestion process used by Europeana

   1. Record counters

   2. Errors in the data ingestion process and details of the errors

   3. Number of records that are considered for ingestion

The numbers show the successful numbers and total numbers of each step. The steps are:

1. Harvest (H): how many records of your total dataset have been imported successfully

2. Validate(edm external(Vi): how many records passed the validation of the external EDM [ Conversion will be added soon for XML files with records that do not comply with the required EDM format before validation.]

3. Transform (T): how many records have been been transformed from the external EDM to the internal EDM [The internal EDM consists of the external EDM and information about the dataset that is added during this import process.]

4. Normalise (N): how many records have been normalised. Normalisation acts on individual values in the data and could include deletion of double spaces and of duplicate values.

5. Enrich (E): How many records have been successfully enriched with the information of processing the data.

6. Process media (M): for how many records the linked media could be found

7. Preview (Pr): how many records were made available in the preview of a copy the Europeana website  (see chapter 7)

8. Publish (Pu): how many records records were published in a copy the Europeana website (see chapter 7)

The colours of each step indicate how successful this step was:

• Green: step completed without errors, all records are considered for ingestion

• Yellow: non critical warning. Problems with the records have been detected, but the records will still be considered for ingestion.

• Red: critical warning. Incomplete records have been detected, these records will no longer be considered for ingestion.

View errors

Click on “view detail(s)” to see the details of the error message(s).

These messages are generated by… [TBC: Incomplete sentence?]

Example above: a record is missing a title or description. EDM documentation is available to look up details mentioned in error messages. These error messages come from the library parsing tool. 

Copy the error messages and/or make a screenshot. Keep the error messages and the dataset close. 

Reviewing steps

You can click on the steps to view the information that you’ve entered in the previous steps by clicking on the icons of the steps or clicking on the “previous step” button. 

You can only view the information, it is not possible to make changes. You can go forward again by clicking on the corresponding icons or by using the forward arrow.

Preview records in Europeana

Click on “view preview” to view your data in a copy of the Europeana website. It can take up to 15 minutes for a preview to be generated. Please wait if your data is not showing yet.

Viewing Tier 0 items

It is possible that not all your items are shown in this view. Records with media Tier 0 are hidden by default. You can make these records visible by clicking on “More filters”, scroll down, click the button “Show only items not meeting our publishing criteria” and click to confirm this filter.