This is and introductory graduate course to cloud computing. The goal is to give a broad overview of cloud computing delivery models and the underlying technology areas, such as datacenter networking, virtualisation, distributed systems, storage and resource management. We will provide an overview of the tools, methods and technologies used by cloud services providers today. The course will involve a set of hands-on exercises such as setting up and using services provided by Amazon, Google, etc.
It is a graduate course and the participants are expected to present on different topics, so called deep dives. We will meet ten time during the spring. Each session will consists of an overview of a technology area and is then followed by deep dive presentations by participants. The course will give 7.5 credits and we will have the first meeting on February 3/2 at 10.15-12.00 in the seminar room at Department of Automatic Control, M-Building. The course is scheduled to be held at that time and locations on Tuesday on somewhat irregular intervals.
The course will to a large extent be based on acadmic papers, white papers, and technical report and we will not follow any course book closely. However, there are two good books that adresses the bigger picture.
Professor Birman has also kindly made his slides available for us to use in the course.
About Us
The course will be given by
Jorn Janneck and
Johan Eker. TA for the course is
Victor Millnert. The way the course is setup each session will consist of an introductory part given by either of us, which is then followed by presentations by the participants. The content of the particpant presentations is designed in collaboration with whoever of the course leaders that is responsible for that particpular session.
Prerequisite
Basic knowledge of system level programming and operating systems.
Examination
- Attend the majority of the sessions.
- Provide at least one student presentation
- Hand in the home assignments
- Course project
Session 1. (Tuesday 3/2) - What Cloud? [Johan Eker] slides
Give an introduction to cloud by going over a number of definitions and concepts. We will take a closer look at the value proposition and evolution of cloud. The fundamental technology components that make up cloud are identified and discussed.
- Different types of cloud?
- Who are the players?
- The heritage: Distributed systems, Grid computing, Cloud computing, etc - more than words?
- What does Netflix look like? How is Amazon's cloud working?
- What is a IaaS, Paas, Saas?
Amazon is developing and running their own technology. They have a wide range of product IaaS such as EC2, CloudFront, S3, Dynamo, CloudWatch, AutoScaling, ELB, etc., and here we take a closer look at their tools. And of course there is Google Compute Engine.
Reading:
- Michael Armbrust et al. "A view of cloud computing". Commun. ACM 53, 4 (April 2010), http://doi.acm.org/10.1145/1721654.1721672
Home assignments: Homework Assignment 1
Session 2. (Tuesday 24/2) - Building Cloud Services [Johan Eker] slides
We will discuss the architecture of cloud services and how they are constructed. Take a closer look on how a search engine may look under the hood. What are the components and design objectives.
- What is a cloud application?
- How does Google work?
- What's is the architecture of Netflix?
- What are the technical components needed for building succesful cloud applications?
Reading:
- Luiz André Barroso, Jeffrey Dean, and Urs Hölzle. "Web Search for a Planet: The Google Cluster Architecture". IEEE Micro 23, 2 (March 2003). http://dx.doi.org/10.1109/MM.2003.1196112
- Jeffrey Dean and Luiz André Barroso. 2013. "The tail at scale". Commun. ACM 56, 2 (February 2013).http://doi.acm.org/10.1145/2408776.2408794
Home assignment: Homework Assignment 2
Session 3. (Tuesday 3/3) - Distributed Computing pt. I [Jorn Janneck] slides
Two lectures to create a basic overview of related topics in distributed computing that form the foundation for the large scale computing units found in cloud. This is a very broad area and deserves not one, but a number of courses in its own right. The topics are selected as being relevant to the current development in cloud computing, i.e. the massive scale and ubiquitous presence, which in turn puts new demands on scalability and availability.
We will start with some classical distributed computing:
- Definition of distributed systems
- Consistency definitions
- Byzantine protocol
- ACID databases
- Time
Student presentation topics
-
Shubhabrata Sen - Logical Clocks - slides
- Lamport, "Time, Clocks and the Ordering of Events in a Distributed System", Communications of the ACM, July 1978 -
- Antonio Franco - Consensus in distributed systems - slides
- Fischer, Lynch and Patterson, "Impossibility of Distributed Consensus with One Faulty Process", J. ACM 32, 2 (April 1985), 374-382. DOI=10.1145/3149.214121 http://doi.acm.org/10.1145/3149.214121
Additional sources:
- Seth Gilbert and Nancy Lynch. "Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services." SIGACT News 33, 2 (June 2002 http://doi.acm.org/10.1145/564585.564601
- Werner Vogels. "Eventually consistent." Commun. ACM 52, 1 (January 2009). http://doi.acm.org/10.1145/1435417.1435432
- Dan Pritchett. "BASE: An Acid Alternative." Queue 6, 3 (May 2008), http://doi.acm.org/10.1145/1394127.1394128
Session 4. (Tuesday 17/3) - Distributed Computing pt. II [Jorn Janneck] slides
We take a look at a selection of algorithms that have received renewed attention for use within massively scalable distributed systems.
Student presentation topics:
- Victor Millnert - The Paxos algorithm - slides
- Chandra et al. "Paxos made live: an engineering perspective". In Proceedings of the twenty-sixth annual ACM symposium on Principles of distributed computing (PODC '07). ACM, New York, NY, USA, 2007. http://doi.acm.org/10.1145/1281100.1281103
- Lamport, "Paxos made Simple", 2001
- Manfred Dellkrantz - Distributed hash tables - slides
- Stoica et al. "Chord: A scalable peer-to-peer lookup service for internet applications". SIGCOMM Comput. Commun. Rev.31, 4 (August 2001). http://doi.acm.org/10.1145/964723.383071
Additional Reading:
- The Bloom filter
- Andrei Broder et al., "Network Applications of Bloom Filters: A Survey", Internet Mathematics
- Test the algorithm with this http://billmill.org/bloomfilter-tutorial/
- Distributed network management ("Self-stabilizing Systems in Spite of Distributed Control", Edsger W. Dijkstra, Communications of the ACM, November 1974]
Session 5. (Tuesday 24/3) - Datacenter Networking [Johan Eker]
We will start by an overview of standard networking technologies such TCP/IP, HTTP, DNS etc. The networking infrastructure of a data center is discussed with a focus on problems related to multi tenancy, isolation, quality of service and management.
Reading:
Student presentations:
- Torgny Holmberg - Datacenter Networkig - Introduction and Cloud Topics - slides
- Harald Gustafsson - Software Defined Networks - slides
-
Diego Kreutz et al "Software-Defined Networking: A Comprehensive Survey", Proceedings of the IEEE 103 (1) , 63. (2015)
Additional reading:
Home assignment: Homework Assignment 3, Homework Assignment 4
Session 6. (Tuesday 14/4) - Virtualization [Johan Eker] slides
Virtualization plays a major role in the cloud stack. In particular the possibility to use cheap commodity x86 servers, despite their lack of virtualization support, accelerated the cloud development. This lecture will give an overview of the principal technologies and platforms in use. We will start by going over some fundamental results from the late 60s and continue with more recent developments.
Reading:
- Gerald J. Popek and Robert P. Goldberg. "Formal requirements for virtualizable third generation architectures." Commun. ACM 17, 7 (July 1974), http://doi.acm.org/10.1145/361011.361073
- J. P. Buzen et al. "The evolution of virtual machine architecture. In Proceedings of the June 4-8, 1973, national computer conference and exposition (AFIPS '73). ACM, New York, NY, USA. http://doi.acm.org/10.1145/1499586.1499667
Student presentations:
- Robert Marklund - Hardware virtualization (Intel) slides
- Adams K, and O. Agesen, "Comparison of Software and Hardware Techniques for x86 Virtualization"
- Christoffer Dall and Jason Nieh, "KVM/ARM: Experiences Building the Linux ARM Hypervisor", Technical report Columbia University.
- Linus Karlsson - State-of-the art containers & hypervisors slides
- LX Containers, Docker, Solaris Zones, FreeBSD Jail, and Docker etc.
- Overview of Currents Hypervisors & VMs
- KVM, Xen, VMWare, etc.
Session 7. (Tuesday 21/4) - Storage [Jorn Janneck]
There is exists a number of different storage approaches for supporting cloud applications: block storage, distributed files systems, databases, key-value stores, object stores. In this lecture will make an overview of the main types and dive deeper into a few of the most well known.
- NAS, SAN, Server SAN
- iSCSI, FC, RDB, RDMA
Reading:
- Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. 2003. "The Google file system". SIGOPS Oper. Syst. Rev. 37, 5 (October 2003)
Student presentations:
- William Tärneberg - Distributed File systems slides
- Konstantin Shvachko et al., "The Hadoop Distributed File System", MSST '10 Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies.
- Sage et al. "Ceph: a scalable, high-performance distributed file system." In Proceedings of the 7th symposium on Operating systems design and implementation (OSDI '06). USENIX Association, Berkeley, CA, USA, 2001
- Structured Data
- James C. Corbett et al. "Spanner: Google's globally-distributed database". In Proceedings of the 10th USENIX conference on Operating Systems Design and Implementation(OSDI'12). USENIX Association, Berkeley, CA, USA, 2012
- The Apache Cassandra data base
- Fay Chang et al. "Bigtable: a distributed storage system for structured data". In Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation - Volume 7 (OSDI '06), Vol. 7. USENIX Association, Berkeley, CA, USA, 2006.
- Memcached
Session 8. (Tuesday 28/4) - Datacenter OS and Applications [Johan Eker] slides
How are applications designed? What are current design principles. We will take a look at OpenStack, which is rapidly becoming the industry standard and its different components Nova, Swift, Cinder, Neutron, Heat.
Reading: Take a look at the docs at OpenStack
Student presentations:
- Jonas Dürango - Cluster resource management slides
- Hindman et al, "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center"
- Shhwarzkopf et al. "Omega: flexible, scalable schedulers for large compute clusters"
- Vavilapalli et al. "Apache Hadoop YARN: Yet Another Resource Negotiator"
- Ola Angelsmark - RESTful acrchitecture (vs SOA) slides
- George Reese, "The REST API Design Handbook", O'Reilley.
- Roy Fieldeing and Richard Taylor, "Principled Design of the Modern Web Architecture", 2002
Session 9. (Tuesday 5/5) - Programming Models [Jorn Janneck]
How to program parallel systems? A walkthrough of MapReduce and the open source Hadoop implementation. And a discussion on typical big data and analytics problems, such as page rank, calculating inverse indexes, recommendation systems, etc.
Student presentations:
- Jens Andersson - MapReduce and more slides
- Jeffrey Dean and Sanjay Ghemawat. 2004. MapReduce: simplified data processing on large clusters. In Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation - Volume 6 (OSDI'04), Vol. 6. USENIX Association, Berkeley, CA, USA
- Hadoop
- Per Persson - Big data problems: social graph, page rank, spam detection slides
Home assignment: Homework #5
Session 10 (Tuesday 12/5) - Datacenter Security [Johan Eker] slides
What role does security play in cloud computing? What are the new (and old) challanges and what are the current trends?
Reading: Thomas Ristenpart et al, "Hey, You, Get Off of My Cloud: Exploring Information Leakage in Third-Party Compute Clouds"
Student presentations:
Additional Reading:
- A Merkle tree ("A Digital Signature Based on a Conventional Encryption Function")
Session 11 - Project Presentations [Jorn Janneck]
TBD