LAWA

To understand what is required to support new innovative Internet applications, a solid understanding of Internet content characteristics (size, distribution, form, structure, evolution, dynamic) is necessary. The LAWA project will build an Internet-based experimental testbed for large-scale data analytics. Its emphasis is on developing a sustainable infrastructure, scalable methods, and easily usable software tools for aggregating, querying, and analyzing heterogeneous data at Internet scale. For decades, compute power and storage have become steadily cheaper, while network speeds, although increasing, have not kept up. The result is that data is becoming increasingly local and thus distributed in nature. It has become necessary to move more analysis to the data, not the reverse.

LAWA will federate distributed FIRE facilities with the rich centralized Web repository of the
European Archive, to create a Virtual Web Observatory and use Web data analytics as a use case study to validate our design. The outcome of our work will enable Web-scale analysis of data, to unlock large-scale study of the content aspect of the Internet and bring this dimension on the roadmap of Future Internet Research. In four work packages we will extend the open-source Hadoop parallel query management software by novel methods for data access and import, develop new methods of distributed storage with indexing, offer scalable aggregation, mine metadata and text along the time dimension, and advance the art of automatic classification of Web contents.

LAWA adds value to the FIRE community by offering access to very large datasets across
thousands of storage and processing nodes, with advanced methods and open-source tools for intelligent analysis at Internet scale, enabling research for the Future Internet to take into account the challenge of content explosion.