Показаны сообщения с ярлыком google. Показать все сообщения
Показаны сообщения с ярлыком google. Показать все сообщения

понедельник, 16 мая 2011 г.

Notes on Google Megastore

Notes on Google Megastore

More notes

Google Megastore

P.S. Megastore is the data engine supporting the Google Application Engine.

вторник, 27 июля 2010 г.

CustomConcurrentHashMap - A framework for concurrent hash map implementations.

com.google.inject.internal.CustomConcurrentHashMap:
"A framework for concurrent hash map implementations."
"Retrieval operations (including java.util.Map.get(java.lang.Object)) generally do not block, so may overlap with update operations (including java.util.Map.put(java.lang.Object, java.lang.Object) and java.util.Map.remove(java.lang.Object))."
"Retrievals reflect the results of the most recently completed update operations holding upon their onset."
"For example, the following strategy emulates the behavior of java.util.concurrent.ConcurrentHashMap: ..."

пятница, 26 марта 2010 г.

The Google File System

The Google File System
Sanjay Ghemawat, Howard Gobioff and Shun-Tak Leung, Google

Abstract
We have designed and implemented the Google File System, a scalable distributed file system for large distributed data-intensive applications. It provides fault tolerance while running on inexpensive commodity hardware, and it delivers high aggregate performance to a large number of clients.
While sharing many of the same goals as previous distributed file systems, our design has been driven by observations of our application workloads and technological environment, both current and anticipated, that reflect a marked departure from some earlier file system assumptions. This has led us to reexamine traditional choices and explore radically different design points.

MapReduce: Simpli ed Data Processing on Large Clusters

MapReduce: Simplified Data Processing on Large Clusters
Jeffrey Dean and Sanjay Ghemawat, Google Inc.

Abstract
MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown in the paper.
Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program's execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.
Our implementation of MapReduce runs on a large cluster of commodity machines and is highly scalable: a typical MapReduce computation processes many terabytes of data on thousands of machines. Programmers find the system easy to use: hundreds of MapReduce programs have been implemented and upwards of one thousand MapReduce jobs are executed on Google's clusters every day.

The Chubby lock service for loosely-coupled distributed systems

The Chubby lock service for loosely-coupled distributed systems.
Mike Burrows, Google Inc.

Abstract
We describe our experiences with the Chubby lock service, which is intended to provide coarse-grained locking as well as reliable (though low-volume) storage for a loosely-coupled distributed system. Chubby provides an interface much like a distributed file system with advisory locks, but the design emphasis is on availability and reliability, as opposed to high performance. Many instances of the service have been used for over a year, with several of them each handling a few tens of thousands of clients concurrently. The paper describes the initial design and expected use, compares it with actual use, and explains how the design had to be modified to accommodate the differences.

Bigtable: A Distributed Storage System for Structured Data

Bigtable: A Distributed Storage System for Structured Data
Fay Chang, Jeffrey Dean, Sanjay Ghemawat, ant others, Google Inc

Abstract
Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.

воскресенье, 25 октября 2009 г.

Guava-libraries from Google

Google опубликовала часть своих библиотек Java, которые они используют в своих проектах, guava-libraries – Project Hosting on Google Code.
Там мало, всего четыре пакета:
Естественно, мой интерес был сконцентрирован вокруг com.google.common.util.concurrent. Порадовало, что там нет практически ничего кардинально нового по сравнению с java.util.concurrent. Только незначительные дополнения. Значит в java.util.concurrent есть практически все даже для разработчиков Google.