Release of libdeltacloud 0.9

I'm pleased to announce the release of libdeltacloud 0.9. libdeltacloud is a library for accessing the Deltacloud API from C or C++ programs. There are several important changes in this release:

• Performance optimizations


• Support for deltacloud driver section


• Support for deltacloud buckets and blobs


• Support for deltacloud loadbalancers


• Full doxygen documentation of all of the functions


• Other minor bugfixes

The release is available from: http://people.redhat.com/clalance/libdeltacloud/0.9

The source code is pushed to the git repository, which lives here: git://git.fedorahosted.org/deltacloud/libdeltacloud.git

As always, questions, comments, problems and patches are welcome!

Release of libdeltacloud 0.8

I'm pleased to announce the release of libdeltacloud version 0.8. libdeltacloud is a library for accessing the Deltacloud API from C or C++ programs. There are several important changes in this release:

• The API for deltacloud_create_instance() has been changed, hopefully for the last time. The previous version of the API had separate parameters for every option we wanted to pass to the deltacloud server. This was confusing and required that we break the API every time we wanted to add a new parameter. The new version of the API expects an array of deltacloud_create_parameter structures that describe the options to pass to the create call. This is more intuitive, more flexible, and shouldn't require us to break the API again.


• We no longer expect return data from POST calls to the deltacloud API. The deltacloud API does not guarantee any return data from a POST call (such as that used to create new instances). libdeltacloud was erroneously assuming it would get data back, which was causing problems with certain deltacloud backends. The new version doesn't expect any return data from POST, and instead requires the user of the library to make an additional GET call if they want additional information about an instance.


• Other minor bugfixes

The release is available here.

The source code is available from: git://git.fedorahosted.org/deltacloud/libdeltacloud.git

As always, questions, comments, problems and patches are welcome!

Release of libdeltacloud 0.7.0

I just pushed out a new release of libdeltacloud v0.7. This version adds the ability to override specific portions of a hardware profile when launching an instance. For example, when launching a cloud instance on GoGrid, you have the choice of several pre-canned instance sizes. However, you can also optionally override the amount of memory while launching the instance, so you get something of a "custom" size. This new libdeltacloud release allows for custom memory, CPU, and storage sizes when launching instances.

FUDCon 2011 - Day 2

Day 2

The second day of FUDCon was much shorter than the first day. In particular,
there were only presentations in the morning, and the afternoon was devoted to hackfests. I started out by going to a few talks.

Asterisk Hacks

This was a fun session for me to attend. I've always wanted to setup Asterisk at home, if for nothing else to play with it. Up until now, I haven't found the time, but I'm hoping to do so in the near future. This talk was about various out-of-the-way features of Asterisk.

Being a telephony system, Asterisk can do all of the normal stuff you associate with telephony: make and receive calls, do call queuing, voicemail, 3-way calling, conference calls, etc. However, there are a number of fun things that Asterisk can do.

The first one is the ability to shift the pitch of your voice. There is a PITCH_SHIFT setting in the Asterisk configuration files that allows you to change the pitch of the voice on the sending or receiving end. There don't seem to be too many practical uses, but it could be fun to be both the secretary and manager of your "company".

The next hack is the ability to avoid someone. Asterisk can do call routing based on the caller ID coming in. You can then have it automatically hang up, redirect to voice mail, or even redirect to another number. All very useful if you are trying to avoid an irate ex-girlfriend ;).

You can also use Asterisk to increase the volume on a connection automatically for certain phone numbers. This might be useful if you often talk to someone that is very soft-spoken.

Finally, Asterisk can be used to trigger calls based on calendar events. It currently has support for integrating with caldev, ical, and exchange calendars, and based on events on those calendars it can trigger a call when that event is starting. This could be very useful for remembering meetings.

GIMP as a pro photo editing tool
I then went to the session on GIMP hosted by María (tatica). She was fun and engaging, and showed off a few tools including GIMP and a couple of others for editing photos. Unfortunately I was deep in the middle of hacking, so while I looked up from time-to-time, I did not fully listen. Sorry María!

Matahari
The last official talk I went to was about Matahari, which is piece of software aiming to provide remote APIs for system management. The full slide deck is here; I'll just go over some highlights.

• Matahari is aiming to provide cross-platform, generically useful APIs over remote interfaces
• Initial targets are for machine/guest introspection and general OS management
• Matahari is implemented as a QMF agent; uses the AMQP protocol underneath
• Matahari does not provide policy; needs to be driven externally
• Initial agents are for Host, Network, Services

This was the last session of the day. The rest of the day was devoted to hackfests. For my part, I spent a bunch of time with Marek Goldmann, the maintainer of Boxgrinder. Boxgrinder and my own project, Oz, share some of the same common goals. The nice part, however, is that there isn't a lot of overlap, and we talked a lot about integrating Oz with boxgrinder and how to speed up operating system installs. I also got quite a bit of hacking done, so I thought this was a useful exercise for me.

All in all, I thought that it was a useful and productive FUDCon. I would definitely go to another one to learn about the things going on in Fedora and meet some more intelligent, fun people.

FUDCon 2011 - Day 1

(sorry this is late; it just took me a couple of weeks to get my act together and get this written)

The weekend of Jan 29 and 30 I went to FUDCon 2011 in Tempe, Arizona. Thanks very much to Robyn Bergeron and all of the rest of the people who did the planning and logistics. This was my first FUDCon, and from my point of view things went off very smoothly.

As has been mentioned elsewhere, the location of the event was excellent. Since I was coming from the snowy northeast, it was great to get a break and have some nice warm weather. The fact that both hotels were in walking distance of the conference venue was also a big plus; I didn't have to rent a car or take a taxi or anything like that.

I arrived on Friday night and headed straight to the primary hotel where the welcome event and opensource.com birthday party were happening. It was nice to meet some people I had only spoken with on IRC, and to have a few drinks and pizza :).

Day 1

On Saturday morning the conference started. My introduction to a "barcamp" style conference started when I showed up at the door to the conference, and I was enlisted to help bring some boxes down to the initial meeting room. People started filtering in, and after some logistics we headed straight into the "pitches" for the talks. Each person who wants to give a talk gives a 20 second synopsis of what they will talk about. Then each of the conference attendees gets to vote for the talks that they want to hear. I pitched my talk about "Cloud Management - Deltacloud, Aeolus, and friends", and after the voting it seemed like there was a lot of interest in the topic. While the votes were being tallied and the scheduled was being finalized, Jared Smith (the Fedora Project Leader) gave his "State of Fedora" talk. At this point we headed over to the main conference venue. My talk was scheduled for the end of the day on Saturday, so I was able to enjoy a few talks before giving mine.

Fedora on ARM

The first talk I attended was the "Fedora on ARM" talk. ARM is one of the world's most widely produced processors, and runs on everything from phones up to servers (in the near future). Why do we want to port Fedora to it? First, the OLPC XO 1.75 is using an ARM processor, and we would like to continue to be able to run Fedora on it. Second, several development boards are currently in production (GuruPlug, Panda board, etc), and running Fedora on those is a great way to get started with ARM development. Third, future tablets and netbooks will likely have ARM processors. And finally, ARM is looking to expand into the server market to compete with x86. Since this is an area that Linux has dominated, it would be good to get a head start here. Servers running ARM aren't quite available yet, but when they do become available they are projected to use 75% less power than equivalent x86 machines.

Previous efforts to port Fedora on ARM have fizzled out. This new effort has a koji build instance at arm.koji.fedoraproject.org, and any developer can submit builds there. It is hoped that this will encourage more participation in the process. The current effort basically has Fedora 13 running on ARM, though there are a couple of problems preventing a full "release". Right now the target platform in ARMv5tel, though they are looking to go to ARMv7 in the future. Now that Fedora 13 mostly works, getting Fedora 14 working is expected to be much easier. All of the patches that have been needed have been sent upstream, and in general things should move a lot faster. In theory, it should be easy to support both ARMv5 and ARMv7 in Fedora (as the instruction sets are compatible), but the reality of the situation is a bit different. ARMv5 has no required FPU, meaning that it is often the case that it has to fall back to software FPU. The ARMv7 specification requires a FPU. Additionally, in ARMv7 some parameters can be passed in FP registers, which obviously will not work on ARMv5. While you can definitely compile for ARMv5 and then run on ARMv7, the performance hit is pretty dramatic. Therefore, there will be two sub-architectures for ARM, one targeting v5 and one targeting v7.

There have been a number of challenges getting ARM building on Fedora. On the hardware side, most of the boards that are available are fairly slow. 512MB of RAM, single-core, with slow I/O seems to be the norm. There are some new "Panda boards" that have 1GB of memory and 2 cores, but the ethernet connection is slow (100Mbit), and the storage subsystem is slow. What this all means is that compiling on these machines takes a long time. On the software side, the challenges have mostly been around software that does not support ARM. For instance, OCaml in Fedora-13 does not support ARM (though this is fixed in Fedora-15). Other packages have had to be patched to deal with ARM alignment problems, compile problems, etc.

In terms of using Fedora on ARM, there are a few hurdles to overcome. The first is that for a random off-the-shelf ARM board, the bootloaders tend to be proprietary. That means that it is not always clear how to load a custom kernel onto the board. The u-boot project can help with many, but not all boards. Even if you have figured out how to load a custom kernel, the Fedora ARM port currently does not provide a packaged kernel, so it is up to you to compile one and create a minimal root filesystem to get the system bootstrapped.

The last part of this session was a quick overview of OLPC. The new version of the OLPC (1.75) will be running an ARMv7 processor, and they want to continue to be able to run Fedora on them. At the same time, there are 2 million x86-based OLPC machines out in the wild, so they want to continue to run Fedora on those machines. In particular, keeping the base distribution small and keeping dependencies down will help on the old OLPCs which only have 256MB of memory.

Sheepdog

The next talk I attended was about Sheepdog, a distributed storage system for Qemu/KVM block storage. I had previously seen a talk about this at KVM Forum 2010, but I went in hoping to understand some more details.

What's the motivation for building Sheepdog? It has a few advantages over other distributed block storage systems. The first is that it is open-source, unlike many of the competitors. The second is that hardware-based distributed block storage systems, like SANs, are very expensive and can be a single point of failure. The third reason is that other open-source distribute block storage systems (like CEPH and Lustre) are complex to setup and administer, and don't always have the performance characteristics expected.

Sheepdog is a fully-symmetric, reliable, scalable, and well-performing distributed block storage system. There is no central node, hence no single point of failure. Instead, blocks are distributed around the machines in a clock-like fashion, so the loss of a single (or in most cases 2) machine does not cause the data to be lost. Sheepdog is intimately tied into Qemu; it uses block layer hooks in Qemu to do its work. On the machine that is running the qemu/kvm process, there is a sheep server process that qemu talks to in order to read and write blocks. When a request for a block that is not on the local node is received, the sheepdog server process reaches out to the other sheepdog process running on other machines and requests the block that it is looking for. Because the data is broken up into 4MB chunks and distributed around the nodes, the performance should be comparable to other network-based storage systems (such as NFS), with much higher reliability.

Sheepdog currently uses the Corosync Engine to do cluster-like operations, which has advantages and disadvantages. The obvious advantage is that Corosync is a well-tested piece of cluster infrastructure that uses a well-known totem ring protocol to do synchronization across nodes. The disadvantage is that corosync is limited to around 48 total nodes. Moving to larger numbers of nodes will require developing a new protocol.

Cloud: The future of computing

I next went to Mike McGrath's talk on the future of cloud computing. The first part of his talk explained the advantages that businesses see in cloud computing (vs. traditional, hardware-hosted-on-premises computing); cost, scalability, and the ability to pay as you go. In this section he also talked about the differences between IaaS, PaaS, and SaaS. Infrastructure as a service (IaaS) is the idea of running virtual machines on hardware that you do not necessarily control. Amazon EC2 is the canonical example of IaaS. Platform as a service (PaaS) is the idea of running your application on a platform that is already prepared for you. Web hosting providers fall into this category. Finally, software as a service (SaaS) is the idea of being completely divorced from the maintenance of the system. Things like Google Docs fall into this category. He then demoed "Libra", an application that he is working on to quickly deploy PaaS applications.

The second part of his talk was how cloud computing is set to replace traditional computing, and how ill-prepared free software is to deal with that new future. In particular, his points are that:

1. HTML5 is set to replace traditional desktop toolkits with a rich interface
2. The new model of cloud computing encourages consumption, not contribution
3. Duplicating a cloud computing model becomes easy (given enough hardware), so there is little incentive for companies to open-source their software stack
4. Current licenses are not prepared for cloud computing
5. The open source crowd will have to compromise to open standards, rather than open-source

While I agree with him on some points, I don't believe that cloud computing is going to completely take over the computing landscape. Computing goes through cycles from thin clients to thick clients, and back again. While cloud computing will almost certainly be a part of the IT landscape for a long time to come, it will be just that: a part. Not every application is suitable for cloud computing. Not every user wants to outsource their concerns to a third party.

Pulp

The last session I went to before my own was about Pulp, which is a software repository management system. It can manage packages, errata, and distros, along with doing repository synchronization from a feed or custom repos.

The architecture looks like:

Essentially, there is a central pulp server that can be interfaced with via REST. It uses a mongodb to hold all of the metadata about packages, errata, etc, and can push (via AMQP) requests to consumer machines. The consumer machines run a small pulp daemon that continually monitors for changes, so if modifications are done outside of pulp, the main server can be notified. The current version of pulp supports modern Fedora along with RHEL-5 and RHEL-6.

When managing repositories with pulp, there are a number of operations available. Repositories can be created, deleted, or updated based on commands from the admin. The data in the repositories can be sync'ed immediately or at a fixed schedule, and the content can come from yum repositories, RHN feeds, or locally created repositories. It also supports global package search across all repos, which may be useful for finding out-of-date packages in the repositories.

The consumers need to subscribe (bind) to certain pulp repositories. Once they are subscribed, the repositories are available through yum, so the normal yum commands will work to install additional packages. The pulp server can do remote installation of packages through the pulp API if desired. As mentioned before, the daemon running on the consumer can keep track of packages that were installed locally, and update the pulp server with that information. Finally, any actions done on the consumer are extensively audited for debugging/security purposes.

What's the difference between Pulp and Spacewalk? Pulp is a much lighter version of Spacewalk, so it can be integrated separately. The hope is to break up Spacewalk into smaller, single-purpose components that are more re-usable (and hence have more users).

Some advanced features of pulp include:

• Batch operations on groups of repositories and groups of consumers
• LDAP integration
• Repository Cloning, so you can start with the same repository but use it for multiple different purposes (think development vs. production)
• A tagging system for consumers
• Full REST API

The future roadmap for Pulp includes:

• Maintenance Windows (do updates at particular dates/times)
• Scheduled installation of packages on consumers
• Support for external content delivery systems to reduce bandwidth
• Clone filters to filter out what packages to clone
• HA and Failover
• Internationalization

Cloud Management
It was then time for my talk. I'm not a great presenter, so I was a bit nervous to start with, but I eventually got into a flow with my talk. I spoke about both Deltacloud and the newly launched Aeolus project, which uses Deltacloud underneath. The outline I used is available here, so I won't go into too much additional detail here. I will say that I had quite a few people come to my presentation, and I had some good questions, so I thought it was successful from that point of view. I'll record some of the questions that I remember below; they make a nice addendum to the material from the talk itself. Any mistakes are mine, since I am working off of my imperfect memory.

Deltacloud
Q: Why use the deltacloud API over something like the EC2 API, which seems to becoming an industry standard?
A: The Deltacloud API is very useful right now, since there is no common API between cloud providers.  While some providers are indeed implementing the EC2 API, it is not ubiquitous.  If it turns out in the future that every cloud that we care about implements the EC2 API, then deltacloud could be viewed as superflous.  Until that point, however, we think that it has significant value in avoid cloud lock-in.

Oz
Q: What about making Oz more configurable, to allow users to upload their own kickstarts?
A: This is a great question, and the answer is that yes, we should make it more configurable. In response to this question, I implemented uploadable autoinstallation files which went into the Oz 0.1.0 release.

Aeolus Conductor
Q: Looking at the full diagram of Aeolus, it is a pretty complicated structure. How are Fedora users expected to install and configure this?
A: This is a valid question, and the answer is not fully formed yet. In order to solve many of the problems that Aeolus is trying to solve, we think the complexity of the internal components is needed. That being said, we really need to make it simple to deploy a simple Aeolus Conductor. To that end, we have a set of scripts called "aeolus-configure" that can install and configure a simple configuration. This certainly needs work, but it is one way in which we can make this more consumable to end-users.

After my talk, there was FUDPub which was a lot of fun. Not much to mention here, other than the combination of drinks, bowling, pool, and friends was fantastic. I would highly recommend doing something similar at future FUDCons.

Deltacloud support in upstream Condor

One of the main tasks that the Aeolus Conductor has to do is to schedule large numbers of cloud instances, possibly across several different cloud providers. In previous projects we've rolled our own sort of scheduler, but instead of re-inventing something for the Conductor, we decided to re-use a known component.

In this case we re-used Condor, a well-known Grid (and other) scheduler. There is good precedent for putting cloud-like functionality into condor; not too long ago, direct EC2 support was added, so using condor in this way was not unprecedented.

We didn't want to use the existing condor EC2 infrastructure, however, since that ties you into a single cloud. Instead we decided to write new code to utilize the Deltacloud API, which will give us nice cross-cloud functionality.

I'm pleased to say that as of January 7, the code to manage cloud instances via deltacloud is in upstream condor. That means that this code will be part of the condor 7.6 release. It is a bit complicated to use, which is why we hide the whole thing behind the conductor. However, it is possible to use the deltacloud support by hand, which I will describe below.

In order to take full advantage of condor's matching and scheduling capabilities, our current solution relies on having provider classads and instance classads. The provider classads describe each of the cloud provider combinations, along with their values. For instance, if you had two AMIs on EC2 that you wanted to be able to use, you would generate two separate classads with all of the data the same except for the AMI id. A provider classad looks something like:

Name="provider_combination_1"
MyType="Machine"
Requirements=true
# Stuff needed to match:
hardwareprofile="large"
image="fedora13"
realm="1"
# Backend info to complete this job:
image_key="ami-ac3fc9c5"
hardwareprofile_key="m1.large"
realm_key="us-east-1a"
provider_url="http://localhost:3002/api"
username="ec2_username"
password="ec2_secret_access_key"
cloud_account_id="1"
keypair="mykeypair"

(pruned for brevity and to protect the innocent). There are a few things to note here. First, in the "stuff needed to match" category are generic things the job classad will match against. That is, the instance classad should not need to know any specifics of the cloud provider backend, just that it wants to run Fedora 13 on a large type instance. Once a provider (or providers) have been found that matches those requirements, the values in the "backend info the complete this job" section are substituted into the actual instance classad, and then the job is submitted.

The jobs themselves then look something like:

universe = grid
executable = job_name
notification = never
requirements = hardwareprofile == "large" && image == "fedora13"
grid_resource = deltacloud $$(provider_url)
DeltacloudUsername = $$(username)
DeltacloudPassword = $$(password)
DeltacloudImageId = $$(image_key)
DeltacloudHardwareProfile = $$(hardwareprofile_key)
DeltacloudKeyname = $$(keypair)
queue

There is a lot going on here, so let's break it down. The first three fields are necessary, but not that interesting; they just give the job a name, tell condor to use the grid universe, and tell condor not to send notifications. The interesting things start happening on the "requirements" line. This is the line that specifies what has to match in order for this job to run. In this case, we are saying that condor gridmanager should look through the provider ads looking for one that has a hardwareprofile that matches "large" and an image that matches "fedora13". These values match what was put in our provider ad above, so condor selects that provider ad. Once it selects that provider ad, it can fill in the rest of the job ad $$ substitutions and submit the job. For instance, in our job ad DeltacloudUsername = $$(username). Once our provider ad has been selected above, that $$(username) gets replaced with "ec2_username". The same happens for the rest of the $$ substitutions.

Once condor has made the match, filled in the values and submitted the job, it will then contact the deltacloud core (from the grid_resource line) to submit the job. If all has gone well to this point, then the instance will actually be launched in the cloud. The instance can also be controlled via condor; if you "condor_rm" the job, then the instance in the cloud will be killed off.

Aeolusproject.org

One of the projects I work on at Red Hat is cloud management. Our cloud projects rely on Deltacloud, a cross-cloud API, to do the real work on the cloud.

Due to some historical baggage, some of the other cloud projects were also under the Deltacloud name although they didn't directly deal with the API. In order to reduce confusion, we have renamed these projects under a new umbrella project, Aeolus. The website for this is http://www.aeolusproject.org. The main project under the Aeolus umbrella is the Conductor, a web-based UI for managing instances in clouds. There are also several other subprojects that help the Conductor do its job, including:

• Oz (an automated guest installation system)
• Image Factory (a project to do installation of guests along with transformation of them to particular cloud backends)
• Image Warehouse (a project to move data from place to place depending on user-specified rules)
• Audrey (a system for configuring cloud instances on-the-fly)

If you have interest in what Red Hat is doing in the cloud, please stop by the website and see what we have to offer. As always, we welcome comments, questions, constructive criticism, and contributions; please send such to the Aeolus mailing list.