Skip to content

Cloud application backup and OpenStack


At work a few months back, was asked to look at backup for OpenStack. Specifically backup of virtual machines running in an OpenStack cloud. My prior project (backup for VMware vCloud) was wrapping up, so the timing was good, and the topic familiar.

Some initial observations.

Before going further, we need to be clear on names…

What AWS calls “instances”, OpenStack calls “instances”, or “servers”, or “hosts”. VMware, VirtualBox and KVM use the term “virtual machines” for the same purpose. In the following, I am going to use “instance” to refer to a virtual machine.

We also need to be clear on the need…

Some argue that a built-for-the-cloud application does not need traditional backup. I agree … mostly. The notion is that we rely on a robust versioned and replicated object store to preserve state, and instances are spun up to handle load, but do not have state to preserve.

For some applications, this model can work. No need for traditional backup.

For more traditional sorts of applications, the cloud model may not apply. There are lots (and lots) of existing working useful applications, built to an older model, that we might choose to spin up in a cloud. Frankly, I suspect there are many rich applications where web-scale is not needed, and the “pure” cloud application model is more trouble than it is worth. For those (many) applications, a more traditional sort of backup is needed.

So there is still a need for the backup of applications (instances) in the cloud.

Note also that storage for backups calls for a different performance profile. Backups are write-mostly, read-seldom, and multiple backups can be enormously compressed by deduplication. This profile differs from other cloud storage services.

In outline, efficient cloud application backup is simple. Note that a cloud application may consist of one or more instances, with one or more volumes (virtual disks) attached to each instance.

Given a list of instances, and a list of associated volumes…

  1. Collect the metadata for the instances.
  2. Quiesce the instances (interrupts application activity).
  3. Snapshot all (wanted) volumes in all instances.
  4. Resume the instances (so as to minimize interruption of service).
  5. Collect changed-block lists for all volumes in all instances.
  6. Store the application/instance metadata and changed blocks into backup storage.
  7. Notify of backup completion.

Note particularly that we want the quiesce/snapshot/resume sequence to be as fast as possible, to minimize any interruption of application service. Also scanning and/or copying multi-gigabyte volumes on every backup is expensive and silly. Smart volume storage can keep track of changed blocks, and enormously accelerate backups.

Note also that this is not a one-instance at a time backup!

Given the above outline, the existing OpenStack APIs for backup are … silly.

  • Cinder allows you to backup a volume … if the volume is not attached to an instance.
  • Nova allows you to backup a single instance … to Glance (the object store)

Add to this: in testing on “stable/icehouse” branch, if I start more than one “nova backup” in succession, only the first succeeds.

The existing OpenStack APIs for backup are not adequate.

Poking around, it looks like all the pieces are present (or nearly) for efficient backup of applications in the cloud, at scale, but … not assembled as a working whole.

The next few months should prove interesting. :)

Coming up to speed with OpenStack


Was asked to look at OpenStack a few months back. Had read a bit prior, and was impressed with the velocity and vigor of this open source cloud platform and project.

Made the mistake of starting with OpenStack. Read a large chunk of the OpenStack documentation. Stood up instances of devstack both on virtual and physical hosts. Not everything made sense.

OpenStack exists because of Amazon Web Services (AWS). Amazon does an excellent job hosting cloud applications, and is the largest player. As a developer, building a cloud application for AWS makes good sense.

Some organizations want to host cloud applications in-house. Hosting an in-house cloud can make sense for a variety of reasons, ranging from performance through cost, control, and legal requirements. If you want to host your own cloud, you want a cloud that works for both existing and newly developed cloud applications. Since cloud application developers are most likely to build for AWS first, you want a cloud as close as practical to AWS.

So OpenStack must be close to AWS from a developers perspective … and is, mostly.

To understand OpenStack, you really need to understand AWS, first. The OpenStack documentation is not bad. The AWS documentation is much better. You can play with AWS very easily, and at very low cost. (Part of the reason AWS is a success is the ease of starting.) Once you are acquainted with AWS, you will find OpenStack easier to understand.

That said, the OpenStack community does at times lose focus.

To be clear, the AWS API can best be viewed as a thin wrapper around Amazon’s cloud services. As AWS evolves, the AWS API evolves. The API is not a set-in-stone independent entity. The Amazon folk are thoughtful in their choices. They change the services and the API carefully, so as to minimize the impact on developers. But the AWS API does change over time.

A few years ago, Mark Shuttleworth (of Ubuntu) made an argument about OpenStack and AWS compatibility. While in general I strongly agree … Mark seems to be arguing for an exact emulation of AWS. I think this is a mistake … but not far from right.

The OpenStack API is a wrapper around services offered by a particular instantiation of cloud services. The services offered might vary somewhat. Different projects and vendors want to plug in differing implementations behind the service APIs. I think this is a good thing. An organization might want to stand up a cloud with emphasis on different aspects of performance, to suit their usage of applications in the cloud.

As a cloud application developer, I like AWS quite a lot. If considering an alternate cloud, the effort on my part to move my application should be as small as practical.

Small but not zero.

If a cloud service has differing characteristics, my use of that cloud may need to be different.

The AWS API is exactly designed for AWS. The AWS API is not designed for cloud services of differing character. As such, the AWS API is not well suited for differing clouds.

There are parts of the AWS API that should be exactly emulated – the S3 service and API is an obvious candidate. As a cloud application developer I am almost certain to make extensive use of S3 (as simply an excellent web-scalable object store).

The other AWS APIs … not sure if I care. For operations I use much less, all I want is well-defined and well-documented simple mappings from AWS APIs to OpenStack usage. If the services are of differing character, I might need to make differing usage. If the differences are small, the work on my part may be small enough I do not care.

So I do not care about exact emulation in all cases. I do care about minimal difference.


If you are trying to understand OpenStack, start by reading up and playing with AWS.

David Keirsey died


This is about missed connections, absurdly close.

Seems that David Keirsey died, about a year ago. He wrote a book on personality types Please Understand Me that might well have changed the course of my life.

Of my own interest, I read rather a lot of books on Psychology, in high school and for a time after. By the time Keirsey’s first edition of “Please Understand Me” came out, I had stopped those readings.

More recently, after what amounts to a prolonged social accident, I started readings around psychology, again. Read the latest of Keirsey’s books, and suddenly the prior confusing social episode seemed to make sense.

The year 1978 was when Keirsey’s first edition book on personality types appeared, and the year I started at the University of California, Irvine (as a third-year student).

My major was in Physics, but one of my “breadth” areas was Psychology. I took the introductory courses but, because my prior readings were essentially in advance of the entire first year, I did not have to study, and did not interact with the other students in the class. After that I took more specialized courses in Psychology, which the general run of students avoided.

The joker here is that Keirsey’s book, and the more recent social accident, suggest that my strongest match in personality might have been among those students of Psychology. Wish I had known…

Add to this Keirsey was at the California State University, Fullerton for quite a long time. That school is within walking distance of the house in Placentia, where I grew up. After 1978 – the time of Keirsey’s first book on the subject, I spent very little time there. But in the time between … I wonder how many times are paths wove near.

If I had happened to take courses there, might we have met?

At University, I met a girl. Later we married, had kids, and … were not remotely suited. I only knew that I had chose poorly, but had not the slightest clue how to choose well. Divorced. Dated for a time, with poor result, then stopped.

On the early social web, ran into a woman – thousands of miles away – to whom emotionally I was very strongly attracted. Made no sense. We seemed very different. Thought it was whack. We ended up talking quite a lot, but I did not understand her personality at all … as I had no prior experience with that sort. I thoroughly fucked up the conversation. Very much a glitch.

Bit later, occurred to me that I really did not understand different sorts of personalities, so started readings on Psychology, again. In Keirsey’s book found my personality sort (easy), found … and started to understand … the glitch-woman’s personality sort. In the part of the book speculating on compatibility, Keirsey thought those two sorts might be the strongest of all matches.


Read his book about a year or so before he died. Would rather we have overlapped a bit more.

Observation, delayed


A regular exercise. Sit in an outdoor patio at a restaurant, for lunch, and watch. There are patterns. Most for which I do not have names. There is something about big purses, and small women, not interesting. High heels worn wobbly, or slow, or fluid. (Reading a book about the Red Queen, which seems applicible in this context.) Low modest heels and clothes. Clear conformity (boring). Awkward mashups. And occasionally a woman walks past whose appearance and dress seem coherent and well done.

Differing combinations.

From psych readings on personality types, and how appearance reflects type, could I be picking up on type?

Then there are times when my mind goes back later and asks … what was that?

Thursday. Eating at an otherwise empty outside patio. A woman walks past, around, and enters the restaurant. Very nice. Very attractive. A sense that her appearance and personality were in concert.

Finished eating. Stood up, looked around, and found that same woman was the only other person on the patio, and she had taken a table a distance directly behind me. Male optimism wants that to mean something, but objectively there is no basis.

Walked across to the coffee shop. In front of me in line, a blonde woman in cutoffs and backless. Very nice. Puzzling. While I enjoy the display … this was different. Seemed somehow very coherent, and very affective.

Later, looked to see where she had gone, and found she took the place closest to the exit. Again, male optimism wanted that to mean something, but … no traction.

Told in hindsight, this is much clearer than at the time.

Great Love Debate, Long Beach


Another random excursion…

Saw an event offered on a MeetUp group: “THE GREAT LOVE DEBATE comes to LONG BEACH!” Title sounds a bit whack. Never been to that venue, so … bought a ticket. No particular qualification needed.

There is a Facebook page … completely disused, aside from the promoters. Also, turns out, there is a website.

Very odd. Lots of girls. Less than half as many guys. From the description:

“100 of Southern California’s most dynamic SINGLE WOMEN. 100 of its most ELIGIBLE BACHELORS.”

Well, there were certainly more than 100 women present. Some possibly very nice. (Dim light, mass setting, not much to go on. Apparently could have gone much earlier, but did not know beforehand.)

Considerably less than half as many men. (What is that?) As to “most ELIGIBLE BACHELORS”, this I very much doubt. A bunch of older guys looking to get “lucky”. Possibly not the brightest. The comments I heard from the guys … er, yuck. Really not comfortable sitting with that group.

Then there was the panel. They all sell something around singles and dating. Nothing wrong with that in itself, perhaps, but … just they all came across as a bit plastic. Bit too daytime TV. Felt like my brain was melting.

Also there was the girl photographer and guy doing video with an outdated rig. Lots of flash photography. Clearly the event was very well documented. Rather intrusive.

Earlier, took a rather nice three hour hike in the hills at midday. (Green hills, clear air, strong sun, vivid colors, great views.) Drove up to LACMA, wandered a bit, then drove down to Long Beach. Took a couple of Monsters to counteract the fatigue from the hike while driving. Had a pot of coffee in the morning, before.

After about an hour, fatigue and massive amount of caffeine in my system kicked in. Felt too weird. Did not have a reason to stay, so left.

Have to admit, this event still bothers me. Why the turnout? I heard about the event via MeetUp and a friend of one of the panelists. But on social media and the web, the only folk posting are the organizers. Seems pretty dead, otherwise.

So what brought the girls? Not from the web, apparently. Some link to TV? (I cancelled cable a couple years back, so that junk is not on my horizon.)

Tagline from the site:
2014 National Tour to answer the question “Why is Everyone Still Single?”

Well, one explanation is that we are single as the orbits of our lives have not brought us close to someone of interest – and it is not obvious what change will work. In the hyper-social early years – high school and college – that part was a lot easier. Over time, as folk pair off, and work is more predominant, things thin out a lot.

Far too much, in fact.

Working at engineering (software) companies, lots more guys than girls, and interesting girls are quickly claimed.
Before my divorce, I chose to live in a great place to raise kids. Not a place for single women, so no luck locally.

Events that cross orbits are a good thing. Rather a lot of rather nice girls at this event. What brought them?
There were a couple of girls … male optimism or actual interest?

Something that needs attention. Not sure exactly what.

Social mapping/accidents


Early evening, sitting alone on a restaurant patio at an outdoor mall, watching girls, looking for patterns. Pretty girl from inside parks in front of me, holds an entire phone conversation. Waitress comes out, asks if we(?) are ready to order.

Group of four girls, one very nice, and smiling slightly, walk past in the direction of a cluster of restaurants. Several minutes later, they walk past again, leaving. Two of the girls are slightly behind and closer, one ahead on the far side of the smiling girl … and watching me.

Not enough time to shop or dine, so … ? Just how aware of watching eyes are a group of girls? And if so, what did they expect?

Friday, went to an event at a hotel in Westwood. Moroccan-themed, performers, and nominally a wine-tasting. Wandered around. Somewhat accidentally ended up at center stage when the performance started. Took pictures. Looking through the pictures later, found one girl who was not watching the performance.

Several, months back at the first event with that group, quite by chance sat on the main path of the fashion show. This is one of their promotional pictures.

A few months back, bought a ticket to a performance with “stage seating”. Ended up … well, on the stage.

At Friday’s event, looked around and found two very nice girls, right behind me. By itself, nothing odd there, since I was in a very good place to watch the performance. Later, thinking back, this pattern repeats.

At lunch, walked into Greek restaurant for lunch. Greeter says: “Table for three?” Look around and find two rather nice, slightly grinning girls, close behind me. Assumed chance, at the time.

Yeh. I still need a clue.

Gender balance



Most recently Tim Bray posted about Moving the Gender Needle relative to the genders in our profession.

Weeks before I was called a “sexist” by a group of guys in a Programming thread on Google+.

At the same time I was discussing sexual harassment with a well-written, well-expressed girl.

I have a very intelligent daughter, who is about to enter college. She has considered tech fields, and could do well easily (her opinion and mine), but at present has a slightly different direction. I raised my kids to make their own choices, and I respect the choices my daughter makes.

Two sides. The genders are not “equal”, in a strict mathematical sense. (I have a lot of math in my background. My degree from school is in Physics.) Of human nature, males and females are not interchangeable. Some differences are rooted in our human nature.

Those differences … still learning what that means.

Exact gender equality is likely impossible. Likely one gender or the other will be more represented in each profession, with exact mapping to the difference in our natures. (I do not pretend to know where that balance lies, or should.)

This factors out the question of gender representation within the software development field into two parts:

  • Part of the difference is due to the difference in the nature of each gender.
  • Part of the difference is due to mis-treatment of the minority by the majority gender.



Last Saturday hiked up to the cloud base. Always wanted to do that. You know when the hilltops are covered in cloud, and you want to hike up into the cloud base? Except … to be exact … that was not quite what happened.

The hike in question.

Planned on a three-hour hike. Had a Secret Dinner that evening, so time was limited.

The problem with hiking up a hill is that I have a hard time stopping short of the summit (turning down a challenge, hard). Need a point somehow significant.

At the three hour point, I could see the next outward bend in the trail above was covered in mist – at the cloud base. Very cool – that made a great point, once reached, where I could turn around.

Except when reached that part of the trail was clear, and the next outward bend was in mist. Onwards … repeated twice(!) more. Finally the trail crossed a saddle, with mist streaming heavily across. This shall be my goal! Except once reached, the air was clear, and the mist was further up the trail.

The rational/deductive part of my mind applied Occam’s rule – the simplest explanation, that the cloud base had simply moved up – was most likely right.

The other side of my mind noted the noted the coincidence, and that cloud chambers (at cloud boundary conditions) are used by Physicists for extremely sensitive experiments. Was I in some strong way perturbing the cloud boundary? The fact that I had no idea how was not proof against my disturbance of that boundary.

An added oddity. Coming down the trail, I saw a white feather suspended, motionless, in the air just out of reach to my right. The slope above was to the left. The ground sloped away on my right. White feathers are a bit unusual in the local wild bits. I assumed the feather was held in some slight upward current of air. As I passed, within a few feet, I felt no current. The feather was just out of reach. (I could have jumped and grabbed the feather, but that would be a jump over empty air…)

Proof of absolutely nothing to the rational, deductive mind. To the imaginative mind … the coincidence was a bit much.