in new years, goals, 2016

I always seem to find myself reflecting upon the year during the holidays - where I was this time last year and where I am now. This year has been pretty good to me overall. I've gone on several adventures, moved to Texas, gotten a new job, started my own company and so much more. I'm super excited that I've kept to this blog going for a whole year - I've always enjoyed writing but I seem to lack the dedication to keep a single project moving. Fortunately, that is not the case for this blog, this is especially fortunate because last year I wrote a post very similar to this one. This allows me the opportunity to compare what I've done this year with what I wanted to do last year. So here it goes!

According to last year's post, I wanted to do these things in 2015:

  • Obtain RHCSA - This did not end up happening, primarily because my situation changed pretty drastically over the course of 2015 - because of this, the RHCSA was not as valuable as it was to me last year.

  • Complete Master's program - This did not happen either. No good reason, just laziness.

  • Learn more virtualization - This definitely happened.

  • Learn Ruby - I would say this one is 50/50 - I definitely know more Ruby than last year, but I am certainly not proficient yet.

  • Work on personal projects - This absolutely happened.

  • Move - I'm writing this from San Antonio, TX where it is 80 degrees in December. Check!

So all in I was able to accomplish a total of 3 (and a half) goals from last year, 60% met, not too bad!

Now, on to what I've actually done this year:

  • Created SpacePanel - SpacePanel is easily the best thing I've ever personally made. I don't mean best as in it was awesome for other people, but best as in I learned so much writing it. I learned about libvirt, KVM, maintaining an actual project with people who care about the project and even a bit about crowdfunding. The knowledge is definitely the best thing to come from the project, but very close runner up was a new position at a Techstars company called Virtkick.

  • Became a developer at Virtkick - Looking back it is amazing that Virtkick and I ever connected. Virtkick learned of me because I made a snide remark on one of their blog posts (saying how SpacePanel rocked and people should use it instead of Virtkick). This got the attention of the founders of Virtkick who apparently admired my dedication to SpacePanel. Shortly after making that comment, Nowaker and Rush reached out to me, told me they liked SpacePanel and wanted me to come work for them in San Antonio. So that is exactly what I did.

  • Left my position at DigitalOcean - After getting a job offer from Virtkick I toiled for days trying to decide what to do. On one hand, I had a great job at an amazing company, but on the other hand: adventure, experience and the chance to do something I love (writing code). I eventually decided to resign from my position at DigitalOcean and move to San Antonio to work with Virtkick.

  • Moved to San Antonio - About a month after I left DigitalOcean and started working at Virtkick remotely April and I packed up all of our stuff in a big Penske truck and began the three day drive to the great state of Texas.

  • Wrote some cool software - Working at Virtkick has been amazing - I wrote code, all day every day. Previously this was something I only did for fun, but the idea of doing it all the time and getting paid for it was incredible. I learned so much throughout the year and I made a super cool new backend system for the company. Said super cool new backend system is now being tested at Bitkumo.

  • Traveled a bit - The Grand Canyon has always been on my bucket list. Right before we moved we were planning on flying out to Arizona to check out the Grand Canyon. Unfortunately, the move was pretty expensive so we had to cancel this trip. Fortunately, not long after getting to San Antonio we were able to visit the Grand Canyon with several friends. Getting there was an adventure - wanting to do the trip as cost effective as possible we did the 16+ hour one-way trip in a single day, spent two days checking our Arizona, and then did another super-not-fun 16+ hour day in the car. But all in, it was super worth it. You can know something is big, you can see pictures of it and see that it is big, but I find it almost impossible to comprehend how big something like the Grand Canyon is until you are standing there before it. It is, in a word, incredible.

  • Started a business - Along with two of my great friends and colleagues Lev and Mike I started a new cloud hosting called Bitkumo. Bitkumo of course uses Virtkick (specifically the backend system that I wrote myself - how cool is that?). The company is still young (about a week old at the time of writing) but things are moving fast and it is awesome.

So I think the tl;dr is that it has been an awesome year. Now that the year is nearing its end, its time to look to the future. That being said, my goals for 2016 are:

  • Grow Bitkumo into a strong, thriving business - I love business, I love strategy and I feel 100% in my element working with Bitkumo. I'm dedicated to making Bitkumo an awesome platform and a great company in 2016.

  • Get my code into production - I've spent most of 2015 working on the aforementioned backend at Virtkick. In 2016 I want to see this code go to production. This would be the first time something I've written is actually used for anything remotely resembling production - that idea is both frightening and very exciting.

  • Learn Ruby/Rails - This didn't completely happen in 2015 so I'm adding it to my goals for 2016. Ruby as a language, and Rails as a framework have a huge following. I firmly believe becoming proficient in RoR will be a good thing for my career overall, so I'm going to make it happen.

  • Continue growing as a developer - The most rewarding thing for me as a developer is seeing my code in use, but a super-close second is seeing my code improve. I love looking at code I wrote a year ago, saying "Wow this is terrible!" and realizing how far I've come. This needs to continue.

  • Continue growing as a person - This means traveling, spending time with my friends and family, learning, and overall learning to live a better healthier life.

I'm looking forward to another great and productive year in 2016. I hope 2015 has been great for you and I hope 2016 is even better!

in Getting Started, node.js, promises, callbacks

Coming from Python, Node.js quickly blew my mind with the whole concept of asychronous stuff. I don't think I actually understood managing things asynchronously for a few months after I started writing Javascript (lets be honest, I probably still don't completely understand it). While the wound is still fresh, I figured I would take some time to write a basic guide for navigating callbacks and promises in Node.js.

Why

If you've come from something like Python or other popular languages threading has likely been something that was either done automatically by a module or something you've had complete control over. In Node threading doesn't really exist, you have one thread so good luck and have fun!

With >1 threads you can do several things (perhaps things that are even blocking) at the same time without impacting the application as a whole (that is, other requests can still be served). With a single thread, a blocking action will block the application as a whole - if you make a synchronous call other requests will need to wait for that task to finish. This is clearly undesirable for things like web applications. What good is an application that can only serve one request at a time? Node addresses this with callbacks, and more recently promises.

Callbacks

A callback is a function that is executed after a task is completed. It usually looks something like this:

asynchronousFunction(argument, function(response) {
	// When the function is finished doing whatever it is doing, this stuff is executed
    // The callback takes any return values as arguments, so if asynchronousFunction returned 1+1, response would be 2
    console.log("asynchronousFunction is done with the response", response);
});

You can also use a callback that is defined elsewhere like this:

asynchronousFunction(argument, callbackFunction);

function callbackFunction(response) {
	// This is executed when the function is done doing its stuff
    console.log("asynchronousFunction is done!");
}

In general, callbacks will first take an error argument and then other arguments that are returned by the function that is utilizing the callback. You can test to see if the error object exists after each call to see if the call was successful:

asynchronousFunction(argument, function(error, response) {
	if (error) { 
    	console.log("Something broke!");
        return; 
    }
    // If this is executed there was no error!
});

The problem with callbacks is they are a little messy. Consider a chain of asynchronous functions that need to run one after another:

aFunction(argument1, function(response) {
	// This runs after aFunction is done
	bFunction(argument2, function(response2) {
    	// This runs after bFunction is done
        cFunction(argument3, function(response3) {
        	// This runs after cFunction is done
        });
    });
});

Not too pretty right? It gets even uglier if you want to deal with errors properly at every step of the chain:

aFunction(argument1, function(error, response) {
	// This runs after aFunction is done
    if (error) { return error; }
	bFunction(argument2, function(error, response2) {
    	// This runs after bFunction is done
        if (error) { return error; }
        cFunction(argument3, function(error, response3) {
        	if (error) { return error; }
        	// This runs after cFunction is done
        });
    });
});

Fortunately promises help make this much neater, easier to read and easier to control.

Promises

A promise is essentially an abstraction of a callback that uses simple to understand syntax and structures. A promise is just that, a promise to give something in the future. For example, say asynchronousFunction from above makes a call to an API requesting all of the virtual machines on an account. That process isn't instant, so it returns a promise which is a placeholder for the list of all the virtual machines on an account. A promise can be either resolved or rejected. A resolved promise is one that fulfilled its promise so to speak and actually has the data you were promised. A rejected promise happens if an error is encountered.

So to rewrite the asynchronousFunction from above using promises you would do something like this:

asynchronousFunction(argument).then(function(response) {
	// Executed after function is done
});

The flow works just like it states, do this function, then do this. You can also pass around promises pretty easily:

function aFunction() {
	return asynchronousFunction(argument);
}

aFunction.then(function(response) {
	// Executed after asynchronousFunction is done
});

Error management is a breeze:

asynchronousFunction(argument).then(function(response) {
	// This is executed if the promise is resolved
}).catch(function(error) {
	// This is executed if the promise is rejected
});

Promises really shine when you have to link a bunch of stuff together. Recall how callbacks looked with multiple functions and then take a look at the syntax used by promises:

aFunction(argument).then(function(response) {
	// Executed after aFunction is done
	return bFunction(argument2);
}).then(function(response2) {
	// Executed after bFunction is done
	return cFunction(argument3);
}).then(function(response3) {
	// Executed after cFunction is done
    return "All done";
});

Much cleaner and easier to read in my opinion. Errors can also propagate up the promise chain fairly easily as long as all promises in the chain are returned properly. A single catch statement for the entire chain will catch any errors thrown while processing those promises:

aFunction(argument).then(function(response) {
	// Executed after aFunction is done
	return bFunction(argument2);
}).then(function(response2) {
	throw new Error("foobar"); 
	// Executed after bFunction is done
	return cFunction(argument3);
}).then(function(response3) {
	// Executed after cFunction is done
    return "All done";
}).catch(function(error) {
	// Caught error from bFunction and stops executing
});

Bluebird is (in my opinion) the best library to use for this sort of thing. You can do tons of really cool stuff with it, including converting things that require callbacks to promises using promisifyAll:

var libraryThatUsesCallbacks = require("callbackLibrary");
var Promise = require("bluebird");

Promise.promisifyAll(libraryThatUsesCallbacks);

// libraryThatUsesCallbacks can now be used like so

libraryThatUsesCallbacks.aFunctionAsync(argument).then(function(response) {
	// I'm a promise now! 
});

// Bluebird adds new functions that basically map to the old function name + async, so bFunction would become bFunctionAsync

This can generally be done for all functions that use the callback(error, arguments) pattern though Bluebird does support other arrangements as well. Bluebird supports a wide array of other things, I would recommend taking a look at the API documentation.

Conclusion

Given the single threaded nature of Node applications learning how to write asynchronous code, and more importantly, think asynchronously, is necessary to becoming a successful Node developer. This isn't an easy task, especially if you are not used to thinking about things in this way, it took me a long time to fully grasp the concepts and properly use asynchronous functions so don't worry if it takes a bit. Hopefully this post will help you learn the basics if you are looking to develop Node applications. Please feel free to leave any questions in the comments.

in libvirt, node, node.js, node-libvirt, promises

After adding a couple things to Node-Libvirt one of the maintainers pointed out my original article on Node-Libvirt referenced the old callback style of using the library so I wanted to take a moment to write up a new guide for using Node-Libvirt with promises rather than callbacks. Since my first post offers a fair amount of detail regarding Node-Libvirt in general, I'll limit this one to showing syntax only.

Hypervisor

First lets take a look at connecting to the hypervisor using promises. Of the changes, this is the most prominent (in my opinion anyway).

Connection

var libvirt = require('libvirt');
var hypervisorObject = libvirt.hypervisor;

var hypervisor = new hypervisorObject('qemu:///system');

hypervisor.connect(function() { 
    // The hypervisor object is now connected. From here you can do just about any call you want...
    // Like hypervisor.lookupDomainById(1);
});

You'll probably want to wrap the code you need to access the hypervisor with in the hypervisor.connect() function to ensure the connection has been made before you attempt to communicate with the hypervisor.

Everything Else

It isn't really worth noting each and every function I mentioned originally, but I will give you a couple examples for how things look using promises.

The basic idea is you add Async to the name of the function you want to call, and then use then instead of a callback. For example:

hypervisor.helloWorld(function(callback) {
	// Old style
});

hypervisor.helloWorldAsync(arguments).then(function(return_value) {
	// New style
});

For real example take a look at a call to getNodeInfo:

hypervisor.getNodeInfoAsync().then(function(nodeStats) {
	// nodeStats now has the return value from getNodeInfo
});

Domains

Domains use pretty much the same syntax shown above but I'll run through a few examples:

// Lets make a new domain
hypervisor.defineAsync(domainXML).then(function(domain) {
	// Use domain to access this domain now
    
    // Now lets restart that domain
    domain.startAsync().then(function() {
    	// Domain has been restarted
    });
});

// Lets lookup a domain with the id 1
hypervisor.lookupDomainByIdAsync(1).then(function(domain) {
	// Use the domain object to do other stuff
});

Would you like to know more?

From here, everything looks almost identical to the syntax you see above. Just append Async to the name of the function and use then instead of a callback and you will be good to go.

If there are any specific calls you are interested in seeing please don't hesitate to leave a comment. Most of these calls do have examples in the Node-Libvirt tests. If you are planning on using Node-Libvirt consider following me on Twitter as I intend to document the full Node-Libvirt API in the future.

Thanks for reading!

in Getting Started, libvirt, node, node.js, domain, hypervisor, node-libvirt

Recently I've been doing a fair amount of work with the libvirt API, specifically using the Node.js bindings found here. The bindings are great and work as expected but I was rather disappointed in the lack of documentation.

The authors reference the tests for the project as documentation, but this can be a bit of a pain to read through especially if you are new to the language. Given this, I'm going to start documenting some of the functions I use most often.

This post assumes you are using the latest version of node-libvirt (>=0.1.3). Please note this is important because earlier versions of this library were not asynchronous - code snippets in this post will not work for earlier versions.

Hypervisor

Before you can do anything else, you must communicate with the hypervisor its self.

Connection

Communication with the hypervisor can be achieved by instantiating a new hypervisor object:

var libvirt = require('libvirt');
var hypervisor_object = new libvirt.Hypervisor(CONNECT_URI);

Connect URI will vary depending on your specific hypervisor (KVM, Xen, etc.) and how you are connecting exactly. For KVM this would be qemu:///system.

Next, you need to actually connect to the hypervisor:

hypervisor_object.connect(function(err) {
	if (!err) {
		console.log("I'm connected!"); 
    }
});

Statistics

Most projects will want to know a bit about the hypervisor, you can gather all sorts of information but I have found the virNodeInfo call the most useful. This provides you with the model of the CPU, memory allocated to the host, number of CPUs, speed of CPUs, nodes, sockets, cores and threads per core. This call is made using the hypervisor object:

hypervisor_object.getNodeInfo(function(err, info) {
	console.log(info); 
});

All remaining hypervisor functions are called in a similar fashion with a callback. Other stats functions of interest include:

  • listDefinedDomains - Returns an array of defined, but not active domain IDs
  • listActiveDomains - Returns an array of active domain names.
  • getNumberOfDefinedDomains - Returns an integer representing the total number of defined but not active domains.
  • getNumberOfActiveDomains - Returns an integer representing the total number of active domains.
  • getNodeFreeMemory - Returns an integer representing the amount of free memory on the host.

From here you can use the hypervisor object to do other things.

Domains

Next, lets get to the meat of the library and discuss domain handling.

Create a Domain

Creating a domain is done using the hypervisor object we created before. You'll need to magic up some XML in the appropriate format to make this work. Once you have that being generated and accessible from Node you can use the following call to make a domain:

hypervisor_object.createDomain(domain_xml, function(err, domain) {
	// Domain is created if no error
});

This will create a non-persistent domain which basically means if the domain is destroyed (completely stopped) it ceases to exist. This is okay if you have all the information you need to reconstruct the domain XML elsewhere (like in a database), if you don't you might want to look into using persistent domains. These can be created using the define function instead of the createDomain wrapper:

hypervisor_object.define(domain_xml, function(err, domain) {
	// Persistent domain created
});

Actions

There are a number of actions available for a domain including:

  • reset
  • reboot
  • suspend
  • resume
  • shutdown
  • start

All of these are called the same way using the domain object. You can get the domain object by accessing domain in the create domain example or you can look a domain up using lookupDomainById or lookupDomainByName, both are covered later.

domain.ACTION(function(err) {
	// If no error action successful
});

Substitute an action from the list above for ACTION.

Get

Getting a domain can be done using either lookupDomainById or lookupDomainByName. Realistically, unless you have a unique application lookupDomainByName will be better used because the domain ID is not static and there is no reliable way of predicting it, the name however is static and should be unique.

hypervisor_object.lookupDomainByName(DOMAIN_NAME_HERE, function(err, domain) {
	// You've got your domain! 
});

Should you need to look up the domain by ID you call that function the same way as shown above.

Destroy

Some people get confused by the nomenclature used by libvirt regarding destroying a domain. In the context of libvirt, destroy does not mean delete in the traditional sense, it essentially means stop the domain now (force it to stop if it cannot be stopped otherwise). This is like pulling the plug on the machine so data loss is a possibility. There is an exception however, if the domain is not persistent destroy will actually remove the domain from libvirt. This is done like the other actions mentioned above:

domain.destroy(function(err) {
	// I'm stopped!
});

If you have yourself a persistent domain and you want to actually make libvirt forget about it completely you would need to use the undefine function:

domain.undefine(function(err) {
	// Poof, I'm gone!
});

Would you like to know more?

I originally started writing this guide with the intention of documenting all the things, but in truth it would be a waste of time. All calls are made using the same basic format:

the_object.WHAT_I_WANT(function(err, RESPONSE_IF_THERE_IS_ONE) {
});

If you find anything particularly challenging regarding hypervisor or domain functions please let me know - I'm happy to help!

While a bit annoying, the tests for node-libvirt are useful if you want to see the context of these calls. You can find the tests for domains here and hypervisors here.

If there is any interest I'd be happy to continue covering the basic calls for other libvirt areas - please let me know if you are interested in that at all.

Thanks for reading!

in project, spacepanel

SpacePanel started as a learning experience primarily - I wanted to learn the in's and out's of virtualization using libvirt. Given the nature of the project, that basic goal was quickly met. However, during the early stages of development SpacePanel received a fair amount of attention from colleagues and general folks across the internet, this inspired me to continue developing SpacePanel. I've never had one of my projects receive any significant amount of attention so I was honestly surprised when SpacePanel was received this way.

Eventually SpacePanel grew to the point I needed help. Lev Lazinskiy, my good friend and former colleague at Linode joined the team to help move development further. The project continued to grow and we eventually reached a point where we simply needed more resources to make SpacePanel bigger, better and hopefully a success. We started an Indiegogo campaign shortly after. Unfortunately, despite lots of hits and encouraging comments we fell significantly short of our fundraising goal. This put Lev and myself in an uncomfortable situation of not knowing what to do next.

Shortly after this we were contacted by the founders of Virtkick. After a bit of discussion with the founders it was decided that instead of trying to go at it alone (and with extremely limited resources) with SpacePanel, I would join their efforts and work on Virtkick exclusively. This started on a trial basis, but recently it was decided that I would continue doing this on a full time basis.

With this being said, it is with a heavy heart that I am officially ending the development of SpacePanel. If you are interested in a control panel like SpacePanel I would highly recommend taking a look at Virtkick. Virtkick is exactly what I eventually wanted SpacePanel to be and I am confident you will enjoy using it.

To those who generously donated to the SpacePanel project, free free to reach out to me at social@spacepanel.io if you would like to reclaim your donation.

In closing, I'd like to run through a few numbers (I like numbers and statistics) concerning SpacePanel:

  • 111 total stars on Github
  • 221 commits spanning a 5 month timeline
  • 58 issues total
  • 30 closed issues
  • 3 milestones met, 1 pending
  • 11 pull requests
  • 2 non-author contributors - KeiroD and deanperry
  • 2 authors
  • ~30 downloads
  • ~70,000 lines of code
  • Countless hours of development time

It's been a fun ride - thank you for those who have helped along the way.