2020 EDITION

Evaluation schemes for Big data and AI Performance of high Business impact (DataBench project sponsored session)

November 4, 2020 - 10:00 - 13:30

For many years companies have been running technical benchmarks to compare the performance of different technologies and systems. This benchmarking process provides very valuable information but in many cases, it is not correlated with the impact of the solutions at business level. This was precisely the starting point of the DataBench project, funded by the EC under the Big Data Value PPP. After three years of intensive work DataBench has created an evaluation framework that is already openly available for the community. The session will introduce the different elements of the framework, including the toolbox, the way it can be used and some concrete examples run with companies and projects that work heavily with data.

DataBench has addressed one of the major barriers to the adoption of Big Data technologies, which has to do with the ability to measure the potential benefits of using Big data technologies in different scenarios. This is not exactly about deciding between one database and another one, but about understanding what the impact of such decision with have on a business process beyond the fulfillment of a technical KPI. This session will allow participants to understand the outcomes produced by DataBench and how to use them in their context through presentations, discussion panels and demos where they will also be entitled to contribute. Attendees will get knowledge on:

The current landscape of Big Data and AI benchmarks
The DataBench framework, which includes a complete set of metrics for the assessment of Big Data technologies
The DataBench toolbox, a web-based tool that provides a unique environment to search, select and deploy big data benchmarking tools, giving the possibility to generate unified technical metrics and derive business KPIs
A comprehensive set of use cases that we have run with companies in different industrial domains and projects of the Big Data PPP to illustrate the way you can get value out of using DataBench
Pipelines and blueprints

The DataBench toolbox follows a platform model and provides a single place where benchmarking communities and users of such benchmarks can meet. It goes beyond any work done so far by translating major technical KPIs into business KPIs, allowing us to establish relationships between technical and business decisions. As such, the session targets a multiplicity of stakeholders: benchmark providers, companies interested in benchmarking big data, projects and members of the big data and AI communities, decision-makers of companies that are thinking about their data-driven transformation and policy makers looking for evidence-based decision-support tools.

On our side, we will bring a comprehensive set of speakers falling precisely under all those categories.

The workshop will be structured around two parts (each of them can be attended independently; like this we want to facilitate that those participants that are not available for the entire workshop, select the most interesting part for them). Attending the complete workshop is in any case recommended to take maximum advantage of the contents.

Detailed agenda

10:00-10:05 Intro. Objectives of the session (Nuria de Lama (Atos)

10:05-10:15 DataBench General Overview (Richard Stevens (IDC, DataBench coordinator)

PART I. Big Data Benchmarking landscape and Big Data Pipelines

10:15-11:15 Session 1. The current landscape of Big Data benchmarks

10:15-10:25: DataBench Framework for Benchmarks, Arne J. Berre, SINTEF
10:25-10:40: Benchmarking platforms and AI, Axel Ngonga, BDVA TF6 Benchmark Lead, University of Paderborn
10:40-10:55: BenchCouncil Big Data and AI Benchmarks, Wanling Gao, Chinese Academy of Sciences
10:55-11:10: MLPerf AI and ABench, Rekha Singhal, Senior Scientist and Head of the Computing Systems-Software Research area at TCS
11:10-11:15: Conclusion on Big Data and AI Benchmarks, Todor Ivanov, LeadConsult

11:15-12:15 Session 2. A Project perspective on Big Data and AI architectural pipelines and benchmarks

11:15-11:20: Introduction to Architectural pipelines, Arne J. Berre, SINTEF
11:20-11:30: I- BiDaaS – Leonidas Kallipolitis, AEGIS
11:30-11:40: TBFY – Brian Elvesæter, SINTEF
11:40-11:50: Track&KNow – Athanasios Koumparos, Vodafone Innovus
11:50-12:00: DataBio – Caj Södergård, VTT
12:00-12:10: DeepHealth – Jon Ander Gómez Adrián, Universitat Politecnica de Valencia
12:10-12:15: Conclusion on Pipelines and related benchmarks, Arne J. Berre, SINTEF

12:15-12:30 Short coffee break to relax and maybe grab a coffee

PART II. Big Data Business Framework and benchmarking tooling support

12:30-13:00 Session 3. The DataBench Business framework: a compelling offering to measure the impact of Big Data Technologies
12.30-12.40: The DataBench business framework, by Gabriella Cattaneo, Erica Spinoni and Chiara Francalanci
12: 40-12.50: The Whirlpool use case, Pierluigi Petrali (Whirlpool)
12.50-13.00: The Fill use case, Harald Sehrschön (FILL)

13:00-13:30 Session 4. A practical journey on how to use the DataBench Toolbox

13:00-13:15 Demo of the DataBench Toolbox (Tomás Pariente, Atos)
13:15-13:25 AI Observatory (Marko Grobelnik, JSI)
13:25-13:30 Fostering adoption of DataBench results. Needs from the point of view of Digital Innovation Hubs (Daniel Sáez, ITI, EUHubs4Data coordinator)

13:30 Concluding Remarks and closing of the session

SPEAKERS OF THIS SESSION