Europeana Cloud - Shared Infrastructure for European Cultural Content
By Pavel Kats, Europeana Development Manager.
Unless you've been living in a cave for the last few years, you've probably noticed that cloud computing technologies are rapidly transforming our lives. Cloud computing, or 'the cloud', for simplicity, have become buzzwords in everyday talk - every kid on the block knows that there are things out there 'in the cloud' and they are supposed to be safer, cheaper and ever accessible with just an internet connection.
To bring the European cultural heritage sector and these recent technological innovations together, the European Commission has sponsored the Europeana Cloud project. Its main goal is to take advantage of recent advances in cloud computing technology and to create a cloud-based digital, trusted infrastructure for aggregating European cultural heritage content. The vision is that adopting the infrastructure will result in an overall economy of scale in IT costs, flexible allocation of computing and storage resources, and improved access to shared metadata and content for both people and compatible tools.
The task of building the infrastructure was assigned to one of the project's work packages (WP2). Three major European aggregators - the Europeana Foundation, the European Library and Poznan Supercomputing and Networking Center - partner with Istituto della Scienza e Tecnologie della Informazione from Pisa, Italy and the Open University, UK. Half a year into the project, the partners have released the first version of the design of the future system and switched to the implementation phase. In this blog post, we'd like to tell you about the essentials of this design, to promote further discussion on it and to encourage cooperation.
The conceptual diagram below features the main components of the future system. At its foundation is a set of data centres, each one set up and administered by one of the institutions. The centres are consolidated into a unified infrastructure which provides database, storage and computation resources for applications and services. At the initial stage, the underlying physical resources of various types, represented in the diagram by colour-coding, will be provided by the founding institutions in accordance with their expertise and IT constraints.
Download a larger version of this diagram
On top of the resource infrastructure, a set of software services will be developed. The services, described in detail in the design deliverable, will be accessible via a set of standardised REST APIs and form a service suite providing basic functionality for aggregation and storage of metadata and content by aggregators. The aggregators will use APIs encapsulating these services to build tools supporting their specialised aggregation flows. The project will develop basic ingestion tools to showcase the potential of the platform and to encourage its adoption by the aggregators. In this vein, the Europeana Cloud infrastructure will be a software-as-a-service (SaaS) platform providing aggregation services and tools, storage used by these services, and computation resources allowing them to run.
Faithful to the spirit of Europeana to promote open-source software, the project will use and develop open-source tools and components. The domain of cloud computing and cloud services has recently seen a huge amount of open-source development and some projects have reached the level of maturity where they can now compete with and beat commercial vendors.
Technology choices made by the infrastructure team to start building the system reflect the decision to be on the forefront of innovation without compromising on maturity of the solutions: Europeana Cloud will use the popular Apache Cassandra distributed database to store metadata and OpenStack, the hottest name in the cloud technology world, for its storage and computation layers. Naturally, the list of used solutions will evolve during the project's lifetime. The software developed by the project will be open-source as well, allowing for easy external contribution by the community and for easy standalone installations exploiting some of the features of the platform.
One of the main themes of cloud computing in general and of Europeana Cloud in particular is cost efficiency for users achieved through consolidation of resources and economy of scale. Institutional and project aggregators will save on IT infrastructure maintenance costs through switching to Europeana Cloud. To sustain the infrastructure beyond the project's lifetime, several sustainability models will be proposed by another part of the project (WP5). Apart from the option to cover the costs directly by paying, institutions will be able to contribute their IT resources to the distributed infrastructure. To ensure the required level of reliability of infrastructure components, a set of SLAs (Service Level Agreements) will be developed and imposed on new joiners.
Another motivation for cloud computing which is going to be explored by the project is flexibility. The idea is that there is no need for constant provision of infrastructure components to support occasional spikes of activity. Instead, these components can be provided temporarily and disposed of when not needed. This allows for flexible pay-as-you-go cost schemes and easier resource planning for organisations. Europeana Cloud will investigate the applicability of such schemes in the cultural heritage domain by maintaining a part of its infrastructure in a public cloud and adding it to the main system as a 'flexible' component which can be extended on-demand to accommodate spikes of activity. A standard use-case envisioned to justify this need are periodical resource-intensive publication cycles which many aggregators maintain to publish new content.
Another important audience which will be addressed by the project are humanities and social science researchers. Despite ubiquitous digital innovation in research and a multitude of new tools being developed for humanity scholars, the usability of these tools is severely limited. Due to many factors, such as incompatible data formats, limited technical or legal accessibility to relevant content, technical hurdles of various kinds, knowledge barriers, and others, there is no free encounter yet between tools and content. The herculean task of enabling this encounter will be addressed by the Europeana Research portal, an extension of the European Library portal, which will be built on top of the Europeana Cloud infrastructure by the project (in WP3).
All in all, Europeana Cloud is a complex technical project with a heterogenous group of stakeholders. We are encouraged by the availability of mature cloud technologies, by overwhelming adoption of these technologies by other industries and by a shared understanding that the time has come to foster this adoption in the cultural heritage sector. It is through the joint effort of different stakeholders - data providers, aggregators, research institutions - that we can work out how cloud technologies will create a revolution in our sector as well.
Get involved
We welcome expressions of interest and proposals of contribution to the project from partners in the Europeana Network and outside it. The first opportunity to discuss all these will be at the Europeana Cloud Architecture Workshop, on 28 October 2013 in the Koninlijke Bibliotheek in the Hague. For inquiries, expressions of interest or any other business please write to the author - pavel.kats@kb.nl.
Tweet Follow @EuropeanaEU