Коллекции от Google помогут не изобретать велосипед. Есть удобные вещи для многопоточности.
Google Collections LIbrary 1.0 final (part1)
Google Collections LIbrary 1.0 final (part2)
P.S. Никогда не пользуйтесь сервисом http://ru.savefrom.net, а для того, чтобы скачать видео, зарегистрируйтесь на youtube.com :)
воскресенье, 28 февраля 2010 г.
четверг, 25 февраля 2010 г.
Cloud vs Grid
The effective use of a Cloud is dependent on the applications. The Cloud can be used appropriately when dealing with linear processes and independent, relatively small data volumes. For applications with larger storage requirements or closely coupled parallel processes with high I/O requirements, Clouds are often useless [here].
Grid works at the "library level" where the users are provided an known "environment" in which their applications should be able to run. In a sense cloud computing offers what grid cannot, a predictable execution environment [here].
Some links from second source:
- Benchmarking Amazon EC2 for High-performance Scientific Computing
- Can Cloud Computing Reach The TOP500?
- HPC in the Cloud
Grid works at the "library level" where the users are provided an known "environment" in which their applications should be able to run. In a sense cloud computing offers what grid cannot, a predictable execution environment [here].
Some links from second source:
- Benchmarking Amazon EC2 for High-performance Scientific Computing
- Can Cloud Computing Reach The TOP500?
- HPC in the Cloud
среда, 24 февраля 2010 г.
3 Phase Commit [3PC]
According to the Wiki - paper "A Formal Model of Crash Recovery in a Distributed System" of Dale Skeen and Michael Stonebraker is main source about 3PC [PDF or PDF].
Paper contains very good analisys and critique of 2PC and definition of 3PC.
There are else some research of thin boundary between Сonsistency and Partition Tolerance (like CAP-Theorem):
"Theorem #2: Rules 1 and 2 are sufficient for designing protocols resilient to a single site failure.
Theorem #3: There exist no protocol using independent recovery that is resilient to arbitrary failures by two sites.
Theorem #4: There exist no protocol resilient to a network partitioning when messages are lost.
Theorem #5: Design Rules 3 and 4 are nesessary and sufficient for making protocols resilient to a partition in a two-site site protocol.
Corollary #6: There exist multisite protocols that are resilient to a simple partition when undeliverable messages are returned to the sender.
Corollary #7: Knowlege of which messages were undelivered at the time the network fails is nesessary and sufficient for recovering from simple partitions.
Theorem #8: There exist no protocol resilient to a multiple partition."
Paper contains very good analisys and critique of 2PC and definition of 3PC.
There are else some research of thin boundary between Сonsistency and Partition Tolerance (like CAP-Theorem):
"Theorem #2: Rules 1 and 2 are sufficient for designing protocols resilient to a single site failure.
Theorem #3: There exist no protocol using independent recovery that is resilient to arbitrary failures by two sites.
Theorem #4: There exist no protocol resilient to a network partitioning when messages are lost.
Theorem #5: Design Rules 3 and 4 are nesessary and sufficient for making protocols resilient to a partition in a two-site site protocol.
Corollary #6: There exist multisite protocols that are resilient to a simple partition when undeliverable messages are returned to the sender.
Corollary #7: Knowlege of which messages were undelivered at the time the network fails is nesessary and sufficient for recovering from simple partitions.
Theorem #8: There exist no protocol resilient to a multiple partition."
BASE: An Acid Alternative
Cтатья. там же есть ссылка на pdf. Из статьи:
--------------------------
Conclusion
Scaling systems to dramatic transaction rates requires a new way of thinking about managing resources. The traditional transactional models are problematic when loads need to be spread across a large number of components. Decoupling the operations and performing them in turn provides for improved availability and scale at the cost of consistency. BASE provides a model for thinking about this decoupling.
--------------------------
Conclusion
Scaling systems to dramatic transaction rates requires a new way of thinking about managing resources. The traditional transactional models are problematic when loads need to be spread across a large number of components. Decoupling the operations and performing them in turn provides for improved availability and scale at the cost of consistency. BASE provides a model for thinking about this decoupling.
вторник, 23 февраля 2010 г.
понедельник, 22 февраля 2010 г.
Расписание семинаров: Вт-18.00
Семинар по Распределенным системам, Транзакциям и Согласованности Данных будет проходить по Вт в 18.00 в ауд 6-52 (начиная с 23 февраля).
В это вторник рассмотрим:
CAP-теорема
Amazon Dynamo
Oracle Coherence
реализация SQL-92 уровней изолированности транзакций в блокировочниках
В это вторник рассмотрим:
CAP-теорема
Amazon Dynamo
Oracle Coherence
реализация SQL-92 уровней изолированности транзакций в блокировочниках
воскресенье, 21 февраля 2010 г.
пятница, 19 февраля 2010 г.
Модель многопоточности.
Уточнение по завтрашнему семинару. Он будет проходить в ауд. 6-56 (или 6-52)
суббота, 13 февраля 2010 г.
Java и языки с динамической типизацией?
Интересная статья об интеграции скриптовых языков в грядущей Java 7.
пятница, 12 февраля 2010 г.
Модель многопоточности.
Мною предложена абстрактная модель частного случая разделения потоками критических секций. Получены результаты по обнаружению т.н. dead-lock'ов достаточно простыми методами. Идеи пока сырые, но все желающие приглашаются их обсудить 20 февраля (суббота) в 11:00 часов в Каразина, аудитория 6-52. Я буду докладываться на семинаре у Григория Николаевича Жолткевича.
Старт семинаров
Предлагается провести первый семинар весеннего семестра во Вторник в !!!18.00!!!.
К этому времени я точно буду в Харькове.
Примерные темы:
- обзор транзакций в Java(JTA, JTS, OTS(Corba)).
- транзакции в Spring + EJB3
- уровни изолированности(READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, READ CONSISTENCY, SNAPSHOT ISOLATION, SERIALIZABLE, CURSOR STABILITY) + феномены(Dirty Write, Dirty Read, Cursor Lost Update, Lost Update, Fuzzy Read, Phantom, Read Skew, Write Skew), три проблемы.
- возможные нетипичные подходы к реализации транзакций (не блоктровочники, не версионники, не оптимистичный ресолв) и феномены/аномалии, которые это может породить.
К этому времени я точно буду в Харькове.
Примерные темы:
- обзор транзакций в Java(JTA, JTS, OTS(Corba)).
- транзакции в Spring + EJB3
- уровни изолированности(READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, READ CONSISTENCY, SNAPSHOT ISOLATION, SERIALIZABLE, CURSOR STABILITY) + феномены(Dirty Write, Dirty Read, Cursor Lost Update, Lost Update, Fuzzy Read, Phantom, Read Skew, Write Skew), три проблемы.
- возможные нетипичные подходы к реализации транзакций (не блоктровочники, не версионники, не оптимистичный ресолв) и феномены/аномалии, которые это может породить.
Anatomy of the Linux ...
Serie: Anatomy of the Linux ...[ALL ARTICLES]:
- Anatomy of Linux process management
- Anatomy of real-time Linux architectures
- Anatomy of Linux synchronization methods
+
All articles from this author:
- Inside the Linux 2.6 Completely Fair Scheduler
- Linux and symmetric multiprocessing
- Inside the Linux scheduler
- Anatomy of Linux process management
- Anatomy of real-time Linux architectures
- Anatomy of Linux synchronization methods
+
All articles from this author:
- Inside the Linux 2.6 Completely Fair Scheduler
- Linux and symmetric multiprocessing
- Inside the Linux scheduler
четверг, 11 февраля 2010 г.
Serializable: Information about the effort to add a true serializable transaction isolation level to PostgreSQL.
Serializable.
Information about the effort to add a true serializable transaction isolation level to PostgreSQL.
* 1 Overview
o 1.1 Serializable and Snapshot Transaction Isolation Levels
o 1.2 Serializable Isolation Implementation Strategies
o 1.3 Predicate Locking
o 1.4 Mixed Isolation Levels
* 2 Current Status
* 3 Development Path
o 3.1 Source Code Management
o 3.2 Implementation
o 3.3 Testing
* 4 R&D Issues
* 5 Discussion
* 6 Publications
"A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[1][2]."
P.S. Впервые нахожу материал как в MVCC-database делают true-serializable.
Information about the effort to add a true serializable transaction isolation level to PostgreSQL.
* 1 Overview
o 1.1 Serializable and Snapshot Transaction Isolation Levels
o 1.2 Serializable Isolation Implementation Strategies
o 1.3 Predicate Locking
o 1.4 Mixed Isolation Levels
* 2 Current Status
* 3 Development Path
o 3.1 Source Code Management
o 3.2 Implementation
o 3.3 Testing
* 4 R&D Issues
* 5 Discussion
* 6 Publications
"A new technique for implementing full serializable isolation in an MVCC database appears in the literature beginning in 2008[1][2]."
P.S. Впервые нахожу материал как в MVCC-database делают true-serializable.
Above the clouds with Android
Эта статья , как мне показалось, заслуживает нашего внимания :)
Summary: The open source Android operating system has taken the world by storm, allowing sophisticated cloud computing applications to run wherever you are. Designed to be highly efficient on battery-powered devices like the T-Mobile G1 smartphone, at heart, Android is Linux®, and there are several layers to the Android programming model that permit the creation of secure applications tailor-made for cloud computing. Soar to new heights with Android and experience mobile computing as you've never experienced it before.
Есть в русском варианте тут
Summary: The open source Android operating system has taken the world by storm, allowing sophisticated cloud computing applications to run wherever you are. Designed to be highly efficient on battery-powered devices like the T-Mobile G1 smartphone, at heart, Android is Linux®, and there are several layers to the Android programming model that permit the creation of secure applications tailor-made for cloud computing. Soar to new heights with Android and experience mobile computing as you've never experienced it before.
Есть в русском варианте тут
Edsger W. Dijkstra Prize in Distributed Computing: Prize winning papers
Edsger W. Dijkstra Prize in Distributed Computing
"Prize winning papers:
* 2009: Joseph Halpern and Yoram Moses for "Knowledge and Common Knowledge in a Distributed Environment" in the Journal of the ACM, 37(3):549-587, July 1990.
* 2008: Baruch Awerbuch and David Peleg for "Sparse Partitions," in Proceedings of the 31st Annual Symposium on Foundations of Computer Science (FOCS), 503-513, October 1990.
* 2007: Cynthia Dwork, Nancy Lynch, and Larry Stockmeyer for "Consensus in the presence of partial synchrony," Journal of the ACM, 35(2):288-323, April 1988.
* 2006: John M. Mellor-Crummey and Michael L. Scott for "Algorithms for scalable synchronization on shared-memory multiprocessors," ACM Transactions on Computer Systems, 9(1):21-65, February 1991.
* 2005: Marshall Pease, Robert Shostak, and Leslie Lamport for "Reaching agreement in the presence of faults," Journal of the ACM, 27(1):228-234, April 1980.
* 2004: R. G. Gallager, P. A. Humblet, and P. M. Spira for "A Distributed Algorithm for Minimum-Weight Spanning Trees", ACM Transactions on Programming Languages and Systems, 5(1):66-77, January 1983.
* 2003: Maurice Herlihy for "Wait-Free Synchronization", ACM Transactions on Programming Languages and Systems, 13(1):124-149, January 1991.
* 2002: Edsger W. Dijkstra for "Self-stabilizing systems in spite of distributed control," Communications of the ACM, 17(11):643-644, November 1974.
* 2001: Michael J. Fischer, Nancy A. Lynch, and Michael S. Paterson for "Impossibility of Distributed Consensus with One Faulty Process," Journal of the ACM, 32(2):374-382, April 1985.
* 2000: Leslie Lamport for "Time, Clocks, and the Ordering of Events in a Distributed System," Communications of the ACM, 21(7):558-565, July 1978."
"Prize winning papers:
* 2009: Joseph Halpern and Yoram Moses for "Knowledge and Common Knowledge in a Distributed Environment" in the Journal of the ACM, 37(3):549-587, July 1990.
* 2008: Baruch Awerbuch and David Peleg for "Sparse Partitions," in Proceedings of the 31st Annual Symposium on Foundations of Computer Science (FOCS), 503-513, October 1990.
* 2007: Cynthia Dwork, Nancy Lynch, and Larry Stockmeyer for "Consensus in the presence of partial synchrony," Journal of the ACM, 35(2):288-323, April 1988.
* 2006: John M. Mellor-Crummey and Michael L. Scott for "Algorithms for scalable synchronization on shared-memory multiprocessors," ACM Transactions on Computer Systems, 9(1):21-65, February 1991.
* 2005: Marshall Pease, Robert Shostak, and Leslie Lamport for "Reaching agreement in the presence of faults," Journal of the ACM, 27(1):228-234, April 1980.
* 2004: R. G. Gallager, P. A. Humblet, and P. M. Spira for "A Distributed Algorithm for Minimum-Weight Spanning Trees", ACM Transactions on Programming Languages and Systems, 5(1):66-77, January 1983.
* 2003: Maurice Herlihy for "Wait-Free Synchronization", ACM Transactions on Programming Languages and Systems, 13(1):124-149, January 1991.
* 2002: Edsger W. Dijkstra for "Self-stabilizing systems in spite of distributed control," Communications of the ACM, 17(11):643-644, November 1974.
* 2001: Michael J. Fischer, Nancy A. Lynch, and Michael S. Paterson for "Impossibility of Distributed Consensus with One Faulty Process," Journal of the ACM, 32(2):374-382, April 1985.
* 2000: Leslie Lamport for "Time, Clocks, and the Ordering of Events in a Distributed System," Communications of the ACM, 21(7):558-565, July 1978."
среда, 10 февраля 2010 г.
CAP Theorem
C == Consistency
A == Availability
P == Tolerance to network Partitions
---
Theorem: You can have at most two of these properties for any shared-data system.
---
Here the keynote speech by Eric Brewer at the ACM Symposium on the Principles of Distributed Computing (PODC) where he fist time introduce such theorem.
Two years later, in 2002, Seth Gilbert and Nancy Lynch of MIT, formally proved Brewer to be correct and thus Brewer's Theorem was born.
Very good disscussion about CAP-Theorem is here.
A == Availability
P == Tolerance to network Partitions
---
Theorem: You can have at most two of these properties for any shared-data system.
---
Here the keynote speech by Eric Brewer at the ACM Symposium on the Principles of Distributed Computing (PODC) where he fist time introduce such theorem.
Two years later, in 2002, Seth Gilbert and Nancy Lynch of MIT, formally proved Brewer to be correct and thus Brewer's Theorem was born.
Very good disscussion about CAP-Theorem is here.
вторник, 9 февраля 2010 г.
Erlang: AXD 301 – ATM switch from Ericsson
Взято тут.
"Как только упоминается язык Erlang, так сразу в качестве козырного туза предъявляется проект AXD 301 – ATM switch (телефонный цифровой коммутатор, если я правильно перевожу это на русский) от Ericsson-а. Поскольку практически всегда, когда речь заходит о нем, приходиться искать в Интернете статьи с указанием объема кода и количества разработчиков, то решил некоторые из них опубликовать здесь. Итак:
AXD 301-A new generation ATM switching system – рекламное-обзорное описание самого устройства. 1998 год.
Ulf Wiger’s Four-fold Increase in Productivity and Quality – презентация с рассказом о том, как программирование на Erlang увеличивает скорость и качество разработки. С примером AXD 301, естественно. 2001 год.
Joe Armstrong’s Concurrency Oriented Programming in Erlang – доклад о конкурентном программировании на Erlang. 2002 год (и тот же доклад, но уже в 2003 году).
Joe Armstrong’s Making reliable distributed systems in the presence of sodware errors – диссертация Армстронга, в которой он говорит, в том числе, и об AXD 301. 2003 год.
Mats Cronqvist’s Troubleshooting a Large Erlang System – небольшая статья о том, как разрабатывался софт для AXD 301. 2004 год.
"Как только упоминается язык Erlang, так сразу в качестве козырного туза предъявляется проект AXD 301 – ATM switch (телефонный цифровой коммутатор, если я правильно перевожу это на русский) от Ericsson-а. Поскольку практически всегда, когда речь заходит о нем, приходиться искать в Интернете статьи с указанием объема кода и количества разработчиков, то решил некоторые из них опубликовать здесь. Итак:
AXD 301-A new generation ATM switching system – рекламное-обзорное описание самого устройства. 1998 год.
Ulf Wiger’s Four-fold Increase in Productivity and Quality – презентация с рассказом о том, как программирование на Erlang увеличивает скорость и качество разработки. С примером AXD 301, естественно. 2001 год.
Joe Armstrong’s Concurrency Oriented Programming in Erlang – доклад о конкурентном программировании на Erlang. 2002 год (и тот же доклад, но уже в 2003 году).
Joe Armstrong’s Making reliable distributed systems in the presence of sodware errors – диссертация Армстронга, в которой он говорит, в том числе, и об AXD 301. 2003 год.
Mats Cronqvist’s Troubleshooting a Large Erlang System – небольшая статья о том, как разрабатывался софт для AXD 301. 2004 год.
Leslie Lamport's book: "Specifying Systems"
Leslie Lamport - one of the Monster in theory of concurrent and distributed systems wrote a book "Specifying Systems" [PDF].
CONTENTS
1. A Little Simple Math
2. Specifying a Simple Clock
3. An Asynchronous Interface
4. A FIFO
5. A Caching Memory
6. Some More Math
7. Writing a Specification: Some Advice
8. Liveness and Fairness
9. Real Time
10. Composing Specifications
11. Advanced Examples
12. The Syntactic Analyzer
13. The TLATeX Typesetter
14. The TLC Model Checker
15. The Syntax of TLA+
16. The Operators of TLA+
17. The Meaning of a Module
18. The Standard Modules
P.S. Book page contains a link on "The Wildfire Verification Challenge Problem".
P.P.S TLA - The Temporal Logic of Actions
CONTENTS
1. A Little Simple Math
2. Specifying a Simple Clock
3. An Asynchronous Interface
4. A FIFO
5. A Caching Memory
6. Some More Math
7. Writing a Specification: Some Advice
8. Liveness and Fairness
9. Real Time
10. Composing Specifications
11. Advanced Examples
12. The Syntactic Analyzer
13. The TLATeX Typesetter
14. The TLC Model Checker
15. The Syntax of TLA+
16. The Operators of TLA+
17. The Meaning of a Module
18. The Standard Modules
P.S. Book page contains a link on "The Wildfire Verification Challenge Problem".
P.P.S TLA - The Temporal Logic of Actions
понедельник, 8 февраля 2010 г.
Конкурс з програмування в середовищі CUDA в Україні
Конкурс з програмування в середовищі CUDA в Україні
Корпорація NVIDIA спільно з українським партнером, компанією Юстар, оголошують про проведення конкурсу по паралельному програмуванню в середовищі CUDA.
Головна мета конкурсу — популяризація CUDA серед українських розробників програмного забезпечення, фахівців і вчених, що використовують паралельне програмування, а також студентів технічних ВУЗів.
Технологія NVIDIA CUDA ™ — це єдина середовище розробки на C, що дозволяє програмістам і розробникам писати програмне забезпечення для вирішення складних обчислювальних задач за менший час завдяки багатоядерної обчислювальної потужності графічних процесорів. У світі вже встановлені мільйони GPU з підтримкою CUDA, і тисячі програмістів вже безкоштовно користуються інструментами CUDA для прискорення додатків – від кодування відео та аудіо до пошуків нафти і газу, моделювання продуктів, медичних зображень та наукових досліджень.
Конкурс відбудеться в період з 15 лютого по 14 травня 2010.
Деталі конкурсу та умови участі на сторінці Конкурс CUDA
P.S. Можно поучаствовать, если придумать какую-нибудь нетривиальную тему:)
Корпорація NVIDIA спільно з українським партнером, компанією Юстар, оголошують про проведення конкурсу по паралельному програмуванню в середовищі CUDA.
Головна мета конкурсу — популяризація CUDA серед українських розробників програмного забезпечення, фахівців і вчених, що використовують паралельне програмування, а також студентів технічних ВУЗів.
Технологія NVIDIA CUDA ™ — це єдина середовище розробки на C, що дозволяє програмістам і розробникам писати програмне забезпечення для вирішення складних обчислювальних задач за менший час завдяки багатоядерної обчислювальної потужності графічних процесорів. У світі вже встановлені мільйони GPU з підтримкою CUDA, і тисячі програмістів вже безкоштовно користуються інструментами CUDA для прискорення додатків – від кодування відео та аудіо до пошуків нафти і газу, моделювання продуктів, медичних зображень та наукових досліджень.
Конкурс відбудеться в період з 15 лютого по 14 травня 2010.
Деталі конкурсу та умови участі на сторінці Конкурс CUDA
P.S. Можно поучаствовать, если придумать какую-нибудь нетривиальную тему:)
11 лютого, Харків: Конференція «Високопродуктивні обчислення та програмування на гібридних суперкомп’ютерних системах на базі TESLA, архітектура CUDA»
Конференція «Високопродуктивні обчислення та програмування на гібридних суперкомп’ютерних системах на базі TESLA ™ від NVIDIA ®, архітектура CUDA», Харків
Дата: 11.02.2010
Место: м.Харків, Інститут сцинтиляційних матеріалів НАН України, проспект Леніна, 60
Захід проходить за сприяння Інституту сцинтиляційних матеріалів НАН України.
Архітектура паралельних обчислень CUDA від NVIDIA дозволила дослідникам усього світу, застосовуючи мову С, задіяти обчислювальні потужності графічних процесорів для вирішення складних завдань високопродуктивних обчислень.
Гібридні суперкомп’ютерні системи на базі NVIDIA Tesla застосовуються для вирішення завдань щодо розрахунків у науково-дослідницькій діяльності, надаючи доступ до нових обчислювальних висот. Рішення постачаються в різних форматах – від персональних суперкомп’ютерів прямо на вашому столі, до гібридних суперкомп’ютерних систем з кластерів з попередньою конфігурацією.
Дата: 11.02.2010
Место: м.Харків, Інститут сцинтиляційних матеріалів НАН України, проспект Леніна, 60
Захід проходить за сприяння Інституту сцинтиляційних матеріалів НАН України.
Архітектура паралельних обчислень CUDA від NVIDIA дозволила дослідникам усього світу, застосовуючи мову С, задіяти обчислювальні потужності графічних процесорів для вирішення складних завдань високопродуктивних обчислень.
Гібридні суперкомп’ютерні системи на базі NVIDIA Tesla застосовуються для вирішення завдань щодо розрахунків у науково-дослідницькій діяльності, надаючи доступ до нових обчислювальних висот. Рішення постачаються в різних форматах – від персональних суперкомп’ютерів прямо на вашому столі, до гібридних суперкомп’ютерних систем з кластерів з попередньою конфігурацією.
ThreadLocal: history of performance improvment
From "Threading lightly, Part 3: Sometimes it's best not to share. Exploiting ThreadLocal to enhance scalability"
ThreadLocal performance
While the concept of a thread-local variable has been around for a long time and is supported by many threading frameworks including the Posix pthreads specification, thread-local support was omitted from the initial Java Threads design and only added in version 1.2 of the Java platform. In many ways, ThreadLocal is still a work in progress; it was rewritten for version 1.3 and again for version 1.4, both times to address performance problems.
In JDK 1.2, ThreadLocal was implemented in a manner very similar to Listing 2, except that a synchronized WeakHashMap was used to store the values instead of a HashMap. (Using WeakHashMap solves the problem of Thread objects not getting garbage collected, at some additional performance cost.) Needless to say, the performance of ThreadLocal was quite poor.
ThreadLocal performance
While the concept of a thread-local variable has been around for a long time and is supported by many threading frameworks including the Posix pthreads specification, thread-local support was omitted from the initial Java Threads design and only added in version 1.2 of the Java platform. In many ways, ThreadLocal is still a work in progress; it was rewritten for version 1.3 and again for version 1.4, both times to address performance problems.
In JDK 1.2, ThreadLocal was implemented in a manner very similar to Listing 2, except that a synchronized WeakHashMap was used to store the values instead of a HashMap. (Using WeakHashMap solves the problem of Thread objects not getting garbage collected, at some additional performance cost.) Needless to say, the performance of ThreadLocal was quite poor.
Dynamo: Amazon’s Highly Available Key-value Store
Dynamo: Amazon’s Highly Available Key-value Store
Abstract
Reliability at massive scale is one of the biggest challenges we face at Amazon.com, one of the largest e-commerce operations in the world; even the slightest outage has significant financial consequences and impacts customer trust. The Amazon.com platform, which provides services for many web sites worldwide, is implemented on top of an infrastructure of tens of thousands of servers and network components located in many datacenters around the world. At this scale, small and large components fail continuously and the way persistent state is managed in the face of these failures drives the reliability and scalability of the software systems.
This paper presents the design and implementation of Dynamo, a highly available key-value storage system that some of Amazon’s core services use to provide an “always-on” experience. To achieve this level of availability, Dynamo sacrifices consistency under certain failure scenarios. It makes extensive use of object versioning and application-assisted conflict resolution in a manner that provides a novel interface for developers to use.
Categories and Subject Descriptors
D.4.2 [Operating Systems]: Storage Management; D.4.5 [Operating Systems]: Reliability; D.4.2 [Operating Systems]: Performance;
General Terms
Algorithms, Management, Measurement, Performance, Design, Reliability.
Abstract
Reliability at massive scale is one of the biggest challenges we face at Amazon.com, one of the largest e-commerce operations in the world; even the slightest outage has significant financial consequences and impacts customer trust. The Amazon.com platform, which provides services for many web sites worldwide, is implemented on top of an infrastructure of tens of thousands of servers and network components located in many datacenters around the world. At this scale, small and large components fail continuously and the way persistent state is managed in the face of these failures drives the reliability and scalability of the software systems.
This paper presents the design and implementation of Dynamo, a highly available key-value storage system that some of Amazon’s core services use to provide an “always-on” experience. To achieve this level of availability, Dynamo sacrifices consistency under certain failure scenarios. It makes extensive use of object versioning and application-assisted conflict resolution in a manner that provides a novel interface for developers to use.
Categories and Subject Descriptors
D.4.2 [Operating Systems]: Storage Management; D.4.5 [Operating Systems]: Reliability; D.4.2 [Operating Systems]: Performance;
General Terms
Algorithms, Management, Measurement, Performance, Design, Reliability.
Group Communication, Total Ordering and Consensus
These are scientific papers which have influenced Postgres-R in some way or another, very interesting readings.
Total Order Broadcast and Multicast Algorithms: Taxonomy and Survey
a good overview, classification and theoretical comparison of group communication systems, by Xavier Défago, André Schipper and Péter Urbán, April 2004
Comparative Performance Analysis of Ordering Strategies in Atomic Broadcast Algorithms
Theoretical analysis of various algorithms for atomic broadcast used for total ordered delivery of messages, by Xavier Défago, André Schiper and Péter Urbán, December 2003
PLATO: Predictive Latency-Aware Total Ordering
An interesting approach to implement low-latency reliable multicasting in datacenters, based on observing arrival times, by Mahesh Balakrishnan, Ken Birman and Amar Phanishayee, October 2006
Ricochet: Lateral Error Correction for Time-Critical Multicast
Ricochet uses IP multicast and lateral error correction to implement reliable multicasting for clustered applications, by Mahesh Balakrishnan, Ken Birman, Amar Phanishayee and Stefan Pleisch, April 2007.
P.S. Взято отсюда.
Total Order Broadcast and Multicast Algorithms: Taxonomy and Survey
a good overview, classification and theoretical comparison of group communication systems, by Xavier Défago, André Schipper and Péter Urbán, April 2004
Comparative Performance Analysis of Ordering Strategies in Atomic Broadcast Algorithms
Theoretical analysis of various algorithms for atomic broadcast used for total ordered delivery of messages, by Xavier Défago, André Schiper and Péter Urbán, December 2003
PLATO: Predictive Latency-Aware Total Ordering
An interesting approach to implement low-latency reliable multicasting in datacenters, based on observing arrival times, by Mahesh Balakrishnan, Ken Birman and Amar Phanishayee, October 2006
Ricochet: Lateral Error Correction for Time-Critical Multicast
Ricochet uses IP multicast and lateral error correction to implement reliable multicasting for clustered applications, by Mahesh Balakrishnan, Ken Birman, Amar Phanishayee and Stefan Pleisch, April 2007.
P.S. Взято отсюда.
Database Replication
These are scientific papers which have influenced Postgres-R in some way or another, very interesting readings.
Don't be lazy, be consistent
The paper that started it all: Don't be lazy, be consistent: Postgres-R, a new way to implement Database Replication, by Bettina Kemme and Gustavo Alonso, September 2000.
Postgres-R(SI)
Combining Replica Control with Concurrency Control based on Snapshot Isolation — a continuation of the Postgres-R idea, making use of MVCC by Shuqing Wu and Bettina Kemme, April 2005.
Database Replication
Lots of other interesting readings about database replicaiton by Bettina Kemme.
Processing Transactions over Optimistic Atomic Broadcast Protocols
Proposes exploiting the spontaneous ordering and using optimistic delivery to decrease the delay induced by the GCS, by Bettina Kemme, Fernando Pedone, Gustavo Alonso and André Schipper.
Pronto: High availability for standard off-the-shelf databases
An eager, update-everywhere replication middleware, using JDBC, by Fernando Pedone and Svend Frolund, 2008
Middleware-based Database Replication: The Gaps Between Theory and Practice
A good overview and comparison of the current state of theory and practice, by Emmanuel Cecchet, George Candea, Anastasia Ailamaki
P.S. Взято отсюда.
Don't be lazy, be consistent
The paper that started it all: Don't be lazy, be consistent: Postgres-R, a new way to implement Database Replication, by Bettina Kemme and Gustavo Alonso, September 2000.
Postgres-R(SI)
Combining Replica Control with Concurrency Control based on Snapshot Isolation — a continuation of the Postgres-R idea, making use of MVCC by Shuqing Wu and Bettina Kemme, April 2005.
Database Replication
Lots of other interesting readings about database replicaiton by Bettina Kemme.
Processing Transactions over Optimistic Atomic Broadcast Protocols
Proposes exploiting the spontaneous ordering and using optimistic delivery to decrease the delay induced by the GCS, by Bettina Kemme, Fernando Pedone, Gustavo Alonso and André Schipper.
Pronto: High availability for standard off-the-shelf databases
An eager, update-everywhere replication middleware, using JDBC, by Fernando Pedone and Svend Frolund, 2008
Middleware-based Database Replication: The Gaps Between Theory and Practice
A good overview and comparison of the current state of theory and practice, by Emmanuel Cecchet, George Candea, Anastasia Ailamaki
P.S. Взято отсюда.
воскресенье, 7 февраля 2010 г.
Postgres-R (8) Architecture
Postgres-R (8) Architecture.
Abstract
Tis document describes the design and architecture of Postgres-R (8), a multi-master replication system for Postgres. It is an extension of the work presented by [KA00]and incorporates enhancements from the subsequent paper Postgres-R (SI) by [WK05]. Further inspiration originates from Slony-II of Neil Conway and Gavin Sherry and from conversation with other fellow hackers of Postgres. Please note that this paper describes the underlying concept and does not necessarily reflect the status of the prototype implementation, which is available from http://www.postgres-r.org. The reader is supposed to be familiar with Postgres internals, especially with Multi-Version Concurrency Control (MVCC) and transaction isolation issues.
Abstract
Tis document describes the design and architecture of Postgres-R (8), a multi-master replication system for Postgres. It is an extension of the work presented by [KA00]and incorporates enhancements from the subsequent paper Postgres-R (SI) by [WK05]. Further inspiration originates from Slony-II of Neil Conway and Gavin Sherry and from conversation with other fellow hackers of Postgres. Please note that this paper describes the underlying concept and does not necessarily reflect the status of the prototype implementation, which is available from http://www.postgres-r.org. The reader is supposed to be familiar with Postgres internals, especially with Multi-Version Concurrency Control (MVCC) and transaction isolation issues.
пятница, 5 февраля 2010 г.
PJP: OSGi: "Listeners Considered Harmful: The “Whiteboard” Pattern"
Listeners Considered Harmful: The “Whiteboard” Pattern.
OSGi - пожалуй самая популярная технология позволяющая устанавливать, заменять и убирать отдельные компоненты ПО в одной JVM БЕЗ ПЕРЕЗАПУСКА ПРИЛОЖЕНИЯ прямо ВО ВРЕМЯ РАБОТЫ ПРИЛОЖЕНИЯ. И как раз патерн Whiteboard - ключевой момент для реализации этих возможностей.
P.S. Взято тут. Для интереса там еще ссылки на несколько pdf-оф.
P.P.S. В конце документа ссылка на Java Tip 79: Interact with garbage collector to avoid memory leaks - борьба с "memory leaks" в Java из-за потери памяти на забытых(не дерегистрированных) listeners. Борьба ведется при помощи weak references.
OSGi - пожалуй самая популярная технология позволяющая устанавливать, заменять и убирать отдельные компоненты ПО в одной JVM БЕЗ ПЕРЕЗАПУСКА ПРИЛОЖЕНИЯ прямо ВО ВРЕМЯ РАБОТЫ ПРИЛОЖЕНИЯ. И как раз патерн Whiteboard - ключевой момент для реализации этих возможностей.
P.S. Взято тут. Для интереса там еще ссылки на несколько pdf-оф.
P.P.S. В конце документа ссылка на Java Tip 79: Interact with garbage collector to avoid memory leaks - борьба с "memory leaks" в Java из-за потери памяти на забытых(не дерегистрированных) listeners. Борьба ведется при помощи weak references.
Подписаться на:
Сообщения (Atom)