Transcript
Page 1: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 1/36

DM2E Interoperability

platform–

Omnom & Pubby

Konstantin Baierer

[email protected]

Page 2: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 2/36

● Components of the DM2E Software Stack● What is Omnom?

– Modelling workflows

– HTTP Protocol

– Provenance

● What is Pubby● Example Pubby setup● Omnom/Pubby Demo

Agenda

Page 3: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 3/36

Omnom and Pubby in DM2E

Page 4: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 4/36

Omnom and Pubby in DM2E

Omnom

Page 5: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 5/36

Omnom and Pubby in DM2E

PubbyOmnom

Page 6: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 6/36

● dm2e-ws– Workflow Engine

– Web Services

● dm2e-gui– Graphical User Interface (Omnom / dm2e-gui)

● dm2e-silk– Linking

● dm2e-pubby– RDF Browser

Components of WP2 Interop Platform

● dm2e-ontologies– Data models/ Validation

● dm2e-direct– Command line ingestion

● dm2e-josso– Single Sign On

Page 7: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 7/36

● dm2e-ws– Workflow Engine

– Web Services

● dm2e-gui– Graphical User Interface (Omnom / dm2e-gui)

● dm2e-silk– Linking

● dm2e-pubby– RDF Browser

Components of WP2 Interop Platform

Omn m

● dm2e-ontologies– Data models/ Validation

● dm2e-direct– Command line ingestion

● dm2e-josso– Single Sign On

Page 8: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 8/36

● dm2e-ws– Workflow Engine

– Web Services

● dm2e-gui– Graphical User Interface (Omnom / dm2e-gui)

● dm2e-silk– Linking

● dm2e-pubby– RDF Browser

Components of WP2 Interop Platform

Omn m

● dm2e-ontologies– Data models/ Validation

● dm2e-direct– Command line ingestion

● dm2e-josso– Single Sign On

Page 9: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 9/36

Omnom: Workflows and Configurations

● Workflow– Abstract description of algorithm

– „Blueprint“ or „Data Wiring“

– Less than a dozen for DM2E

● Configuration– Instantiation of a workflow with concrete values

– One for every ingestion

Page 10: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 10/36

Omnom: What's in a Workflow?

● Typed Input and Output Parameters● Positions wrapping Web Services

Input and Output Parameters

Iterative positions for repeatable output● Connectors wiring the parameters

=> Define data flow

Page 11: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 11/36

Simple XSLT Publish Workflow→

Page 12: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 12/36

Simple XSLT Publish Workflow→

Workflow Input Workflow Output

Page 13: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 13/36

Simple XSLT Publish Workflow→

Workflow Input Workflow Output

Input

Web Service

Output

Position 1

Input Web Service

OutputPosition 2

Page 14: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 14/36

Simple XSLT Publish Workflow→

Workflow Input Workflow Output

Input

Web Service

Output

Position 1

Input Web Service

OutputPosition 2

Connectors

Page 15: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 15/36

Omnom: What's in a configuration?

● Parameter Values– Resolvable URLs

– Strings

● Examples– URL of result RDF/XML

– URL of input XSLT

– Name of a dataset for URI generation

– Name-Value pair for XSLT parameters

Page 16: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 16/36

Web services: Protocol

● WS describe themselves in RDF:– Input / Output parameters

– Human readable description / label

● WS expect configuration in RDF:– Concrete values for input parameters

– Either resolvable URLs or Literals

● WS immediately return a Job in RDF:– Asynchronously work in the background

– Write (intermediate) results to the Job resource

Page 17: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 17/36

Webservice: Execution

Page 18: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 18/36

Convention for description/thing Accept conflicts for RDF Inf. Res.

● RDF/XML at http://mydomain/file/123– Document: Output of a transformation

– Metadata: File Size, Creator, MD5 ...

● HTTP Accept: application/rdf+xml– Return the document ?!

– Return the metadata ?!

● Our solution:– No Accept / application/octet-stream => Data

– Accept: RDF/JSON => Metadata

Page 19: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 19/36

Types of Services

● Backend Services– Workflow Service

– Configuration Service

– Job Service

– File Services● DM2E File Service● MINT

● Web Services to be used in workflows

Page 20: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 20/36

Web Services for Workflows

● Transformation– Transform XML with XSLT to RDF/XML

– Transform RDF with SPARQL to RDF

● Publishing– With provenance to Triplestore

● Iteration– Feed every contained resource to another web service

– ZIP of XML files, OAI-PMH harvesting

● Execute a workflow → Nested Workflows

Page 21: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 21/36

More complex: ZIP XSLT-ZIP Publish→ →

Page 22: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 22/36

Even more complex: Nested Workflows

Page 23: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 23/36

Provenance in DM2E

● All Omnom resources are described in RDF● All results are accesible on the Web

– Either Directly in RDF

– Or as Web Documents, with RDF metadata

● All data is in versioned datasets● [ Dm2e-pubby offers version-aware API ] ● Consumers should "Follow their nose"

Page 24: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 24/36

Provenance: PROV Omnom↔

Page 25: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 25/36

Omnom Web User Interface

● Single-Page Application● BackboneJS

– MVC framework

– Client/Server: Same data representation: JSON-LD (ish, kind of)

● Bootstrap– Looks like Twitter

– Sane Defaults

Page 26: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 26/36

Pubby

● Frontend for Linked Data● Make triplestores web-accesible

– Dereferenceable URI

– 303 Redirects

– Content-Negotiation

● HTML interface

Page 27: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 27/36

DM2E Extensions to Pubby

● CONSTRUCT instead of DESCRIBE– Support Named Graphs

– Configuration defines the relevant subgraph that describes a URI

– Multiple Resources per output graph

● More flexible URIs– 'data' and 'page' configurable

– Dataset-specific URIs

– Regex Matching in URIs

● Generate thumbnails

Page 28: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 28/36

Example Pubby Use Case

● Want to expose library data– Book data from triplestore 1

– Subject data from triplestore 2

● Using the same consistent interface● Full example here

http://data.dm2e.eu/pubby-example

Page 29: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 29/36

Omnom: Workflow Edit

Page 30: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 30/36

Configuration Edit

Page 31: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 31/36

Job Page

Page 32: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 32/36

Pubby View of the result

Page 34: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 34/36

Omnom: Next steps

● Performance● UI/UX improvements● Easier deployment

Page 35: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 35/36

Pubby: Next steps

● Handle weird URIs more gracefully● Properly handle blank nodes

– Collection Ontology Lists

● Caching (hard)● Re-Merge with upstream Pubby

Page 36: DM2E Interoperability platform – Omnom & Pubby (Konstantin Baierer)

2014-03-18 Konstantin Baierer: Omnom & Pubby 36/36

Thank You!

● Demos:– http://omnom.dm2e.eu

– http://data.dm2e.eu/data/

● Source Code / Issue Tracker– http://github.com/DM2E/dm2e-ws

– http://github.com/DM2E/dm2e-gui

– http://github.com/DM2E/pubby

● Get in touch:– [email protected]

– http://dm2e.eu, @DM2Europeana


Top Related