OpenStack SDK Post-Summit Update

This is a long post about the OpenStack SDK. It even has a Table of Contents.

Current Project Status

The OpenStack SDK is quickly heading toward being usable for application developers. Leading up to the OpenStack Summit we had a reasonably complete Resource layer and had been working on building out a higher-level interface, as exposed through the Connection class. As of now, first cuts of a high-level interface have implementations in Gerrit for most of the official programs, and we're working to iterate on what we have in there right now before expanding further. We also had an impromptu design session on Thursday to cover a couple of things we'll need to work through.

Project Architecture

At the lowest level, the authentication, session, and transport pieces have been rounded out and we've been building on them for a while now. These were some of the first building blocks, and having a reasonably common approach that multiple service libraries could build on is one of the project goals.

Session objects are constructed atop Authenticators and Transports. They get tokens from the Authenticator to insert into your headers, get endpoints to build up complete URLs, and make HTTP requests on the Transport, which itself is built on top of requests and handles all things inbound and outbound from the REST APIs.

http://i.imgur.com/A2U6yc4.png

Poorly drawn version of what we're doing

On top of that lies the Resource layer, a base class implemented in openstack/resource.py, which aims to be a 1-1 representation of the requests or responses the REST APIs are dealing with. For example, the Server class in openstack/compute/v2/server.py inherits from Resource and maps to the inputs and outputs of the compute service's /servers endpoint. That Server object contains attributes of type openstack.resource.prop, which is a class that maps server-communicated values, such as mapping the accessIPv4 response body value to an attribute called access_ipv4. This serves two purposes: one is that it's a place we can bring consistency to the library when it comes to naming, and two is that props have a type argument that allows for minimal client-side validation on request values.

Resource objects are slightly raw to work with directly. They require you to maintain your own session (it's the first argument of Resource methods), and they typically only support our thin wrappers around HTTP verbs. Server.create will take your session and then make a POST request populated with the props you set on your object.

On top of the Resource layer is the Connection class, which forms our high-level layer. Connection objects, from openstack/connection.py, tie together our core pieces - authentication and transport within a session - and expose namespaces that allow you to work with OpenStack services from one place. This high-level layer is implemented via Proxy classes inside of each service's versioned namespace, in their _proxy.py module.

Right now many of these Proxy implementations are up for review in Gerrit, but openstack.compute.list_flavors is currently available in master. It builds on the openstack.compute.v2.flavor Resource, simply calling its list method inside list_flavors and passing on the Session that compute was initialized with.

What the high-level looks like

There are a bunch of example scripts in the works in the Gerrit reviews, but some of what we're working on looks like the following.

Create a container and object in object storage:

from openstack import connection
conn = connection.Connection(auth_url="https://myopenstack:5000/v3",
                             user_name="me", password="secret", ...)
cnt = conn.object_store.create_container("my_container")
ob = conn.object_store.create_object(container=cnt, name="my_obj",
                                     data="Hello, world!")

Create a server with a keypair:

from openstack import connection
conn = connection.Connection(auth_url="https://myopenstack:5000/v3",
                             user_name="me", password="secret", ...)
args = {
    "name": "my_server",
    "flavorRef": "big",
    "imageRef": "asdf-1234-qwer-5678",
    "key_name": "my_ssh_key",
}
server = conn.compute.create_server(**args)
servers = conn.compute.list_servers()

Where we're going

General momentum has carried us into this Connection/Proxy layer, where we have initial revisions of a number of services, and by default, we'll just keep pushing on this layer. I expect we'll iterate on how we want this layer to look, hopefully with input from people outside of the regular contributors. Outside of that, results from conversations at the Summit will drive a couple of topics.

  1. We need to figure out our story when it comes to versioning APIs at the high level. Resource classes are under versioned namespaces, and even the Proxy classes that implement the high level are within the same versioned namespace, but we currently expose high level objects through the Connection without a version, as seen in the above examples.

    On one hand, it's pretty nice to not have to think about versions for APIs that only have a v1, but that won't last. Along with that, we're working in a dynamic language on growing APIs. Not pinning to a version of the interface is going to result in a world of pain for users.

  2. We need to think about going even higher level than what we have now. Monty Taylor's shade library came up both at his "User Experience, SDKs" design session, as well as during the impromptu OpenStack SDK session we had, and once we get more of the Connection level figured out, we're going to look at how we can tackle compound operations.

  3. Docs, docs, docs. Terry Howe has been putting in a lot of work on building up documentation, and now that we're moving along more smoothly up the stack, I think we'll soon hit the point where code changes will require doc changes.

    I'm also working up a "Getting Started" guide for the project, as we have some people interested in contributing to the project. Thursday's python-swiftclient session ended in that team being interested in shifting their efforts to this SDK, so we need to make sure they can easily get going and help improve the client and tool landscape.

    For the time being, doc builds will appear at http://python-openstacksdk.readthedocs.org/

  4. PyPI releases. Terry put together a version of the package that could reproduce the examples we showed in our talk on Monday, comprised of master plus a couple of his in-flight reviews for compute and network and mine for object store. As we progress and want to try things out, and to enable people to try along with us, we'll probably keep cutting more releases under 0.1.:

    pip install python-openstacksdk
    

    Keep in mind this is absolutely a work-in-progress, and API stability isn't yet a thing, so check it out and let us know what you think, but don't build your business on it.

  5. Need to get back into some of the administrivia that we've been avoiding recently in the name of expanding the Resource layer. The wiki page could use a refresh to reflect where we're at and what's going on. We need to start using more blueprints and the issue tracker, especially as more people become interested in joining the project. We were able to work without most of that when it was just a couple of us wanting to get this off the ground, but we need to make better use of the tools around us.


Overall, the SDK is coming along nicely. We had some good talks at the Summit and got a lot of interest from people and projects, so the coming months should be another good period of growth for us.

Summit Presentation with Terry Howe

On Monday, Terry Howe and I presented "Getting Started with the OpenStack SDK", a 40 minute talk on why we're doing this, how we're doing it, and where the project is going. Both of us had presented at conferences before, but never jointly, so it was an interesting first time experience, and it seemed to work well. The general gist is that I covered the most of the "why" and "where", and Terry covered most of the "how".

The first half focuses on three key ideas that brought this SDK to being: fragmentation, duplication, and inconsistency in the library and tooling landscape around OpenStack. I dove into each of those areas with examples of why they're an issue, such as how many different clients there are, and how different it can be to work with them. From there I covered some of the goals we have while trying to improve those issues, such as building solid foundations and providing consistent user interfaces.

The second half focuses on showing where we're at and what can be done. Terry took a working example that creates a network, sets up various security group rules, starts up a server, attaches a floating IP, and results in a running Jenkins server. After that, he dove into some of the internals, showing how session, transport, and authenticator work together, and explaining the resource and proxy levels.

After we were done, we had a good 10 minutes of questions, and about another 20 minutes of conversation in the hall afterward. A university professor came up to me to say he wants to use the SDK with his students, which was awesome to hear.

Check out the video here - 42 minutes total.

SDK Conversations at the Summit

In the Marketplace

While spending most of Monday through Wednesday in the Rackspace booth in the marketplace, I talked to a lot of people about the SDK project. It's fun to give away t-shirts and raffle off prizes at conferences, but I'm there to talk with people about the experiences they have with Rackspace, OpenStack, and other platforms, and to advocate for the first two.

I've gotten the SDK "elevator pitch" down fairly well by now for when people turn around and ask what I do. The good thing is that no one thought it was a bad idea! People were excited over various parts of it, mostly between reducing the fragmentation by offering all of the libraries from one package, and a lot were excited about coming up with more consistent interfaces across services.

Overall it was a lot of small conversations that ended with a smile that we're both doing fun stuff and it's all getting better.

Impromptu Design Session

Although we didn't have a session on the schedule, we created one of our own Thursday morning in the Le Meridien lobby. Dean Troyer, Jamie Lennox, Terry Howe, Ken Perkins, and myself gathered to talk for about 40 minutes on where we're going. We talked about two main points: an even higher level than we currently provide, and our multi-version story.

Even Higher Level

Currently we provide an abstraction that gets a user to the point where they can call, e.g., object_store.list_containers(), and they'll receive a list of containers. We've taken care of the lower-level plumbing bits like authentication, session, and transport within the Connection class, which exposes the object_store namespace, containing the higher-level view on top of the account and container resource level.

It was mentioned during this session, and during Monty Taylor's user experience session, that Monty is working on a project called Shade. Shade flies at a higher level where you say give me a working server and it does what's necessary to make that happen. The tool aims to abstract away provider differences in order to complete the task, such as how Rackspace gives you a VM with a publicly accessible IP and HP VMs need to be added to a network and have a floating IP attached to them.

"Give me a server" is a pretty common first step for newcomers, so that's an obvious starting place. "Upload this directory to object storage" is another. If you have others, we'd love to know, and we'd love help to implement them. With where we're working right now, we're not yet on to provider specific plugins, so high-level multistep tasks on vanilla OpenStack are what we're looking for.

Multiversion APIs

At the high level within openstack.Connection, we're not currently making any attempt to expose multiple versions of a service's API. We support authenticating via either a v2 or v3 Keystone, and we support multiple versions of APIs at the resource layer, but you end up with high-level access to a set of unversioned service APIs. On one hand, that makes it fairly nice to work with methods on openstack.object_store, especially since there is currently only a v1 API, but should that actually have a v1 somewhere in there?

A point was brought up that we pin versions in other places, such as our requirements. We couldn't have an unversioned dependency in requirements.txt and expect our code to continue working against its APIs forever. When they go from v1 to v2, things will be different and potentially affect what we've coded against. If you've written against the v1 API, you probably want to stick with it until you've written and tested against the v2 API. As much as the unversioned namespace may feel more friendly, it's eventually going to cause pain.

The "Improving python-swiftclient" Design Session

On Thursday, John Dickinson held a session on how to improve the python-swiftclient project. I'm not a contributor there, but was interested to see what they were planning to do and maybe chime in on getting a few more eyes on the SDK, especially since I threw together a high-level Swift view.

Within the first few minutes, the bulleted list that the group had come up with looked a lot like the bulleted lists we came up with to start the SDK project. They have a lot of work they want to be doing, and we're already on our way doing much of the same. Dean Troyer beat me to the punch of grabbing a mic and asking if it's possibile to put some of these efforts behind both OpenStackClient and the SDK.

Dean and I then gave very quick talks on where OSC and SDK fit in to what they were aiming to accomplish. From there, the conversation shifted towards 'Can we accomplish this over there?' and 'Do we want to accomplish this over there?' The answer to both turned out to be 'yes'.

Coming out of this meeting, we're going to have to quickly bulk up our documentation of the lower-level parts so we can bring these folks up to speed, as one of the first topics was their HTTPConnection class, and the second was from Jamie Lennox on using Keystone's sessions.

We're also going to need to bulk up on a "Getting Started" guide for new contributors coming out of this session and a few other talks I've had. Welcome everyone!


If you got this far, wow. See me at a conference some time for a high five.