Showing posts tagged with development

in Getting Started, development, bookshelf.js, node, node.js, guide

I recently started a project where I was tasked to use Bookshelf.js to interact with the database. The library its self is pretty handy, it is built on top of Knex.js, a great query builder that works with Postgres, MySQL, and SQLite, but my major gripe was an overall lack of examples in the documentation. That being said, I wanted to document what I learned for folks using Bookshelf.js in the future (and for myself when I inevitably forget how it was done).

Installing Bookshelf.js & Knex.js

Installation requires Bookshelf.js, Knex.js as well as the corresponding package for the database server you are using (MySQL, Postgres, SQLite):

npm install knex --save  
npm install bookshelf --save  

Next, install the package for your database server:

// Pick one
npm install pg --save // Postgres  
npm install mysql --save // MySQL  
npm install mariasql --save // MariaDB  
npm install sqlite3 --save // SQLite  

Connecting with Database

Once Establishing a connection with the database is fairly simple, pass in a JSON object of the necessary options:

var knex = require('knex')({  
    client: '{pg|mysql|sqlite3}',
    connection: {
        host: 'host' // IP or domain name
        user: 'user' // DB username
        password: 'password' // DB password
        database: 'database' // DB name
        charset: 'utf8' // Or your preferred charset
    }
});

If you are using SQLite, add the filename directive to the connection directives, this should be the path to your database on the filesystem.

Next, bring Bookshelf into the loop:

var bookshelf = require('bookshelf')(knex);  

With that, you're ready to get started!

Models

The first thing you'll want to do is extend Model so you can interact with tables in your database. This is done like so:

var model = bookshelf.Model.extend({  
    tableName: "nameOfTable"
});

You can now use the variable model to interact with this table.

A quick side note, you can append functions to the model delcaration if need be as shown in this example from bookshelfjs.org:

var checkit  = require('checkit');  
var Promise  = require('bluebird');  
var bcrypt   = Promise.promisifyAll(require('bcrypt'));

var Customer = bookshelf.Model.extend({

  initialize: function() {
    this.on('saving', this.validateSave);
  },

  validateSave: function() {
    return checkit(rules).run(this.attributes);
  },

  account: function() {
    return this.belongsTo(Account);
  },

}, {

  login: Promise.method(function(email, password) {
    if (!email || !password) throw new Error('Email and password are both required');
    return new this({email: email.toLowerCase().trim()}).fetch({require: true}).tap(function(customer) {
      return bcrypt.compareAsync(customer.get('password'), password);
    });
  })

});

Customer.login(email, password)  
  .then(function(customer) {
    res.json(customer.omit('password'));
  }).catch(Customer.NotFoundError, function() {
    res.json(400, {error: email + ' not found'});
  }).catch(function(err) {
    console.error(err);
  });

Creating a New Record

Next, lets create a record - I'll continue using the variable model as declared in the previous section:

new model({  
    'column1': 'columnValue',
    'column2': 'columnValue'
    // and so on...
}).save().then(function(newRow) {
    console.log(newRow.id); // Returns ID of new row
    // Code executed after row successfully created
}).catch(function(err) {
    // Handle errors
});

You can also use forge instead of the new notation:

model.forge({  
    'column1': 'columnValue'
    // and so on...
}).save().then(function(newRow) {
    console.log(newRow.id); // Returns ID of new row
}).catch(function(err) {
    // Handle errors
});

Bookshelf has built in promisification so be sure to take advantage of that feature in your code!

Update Record(s)

Updating records looks pretty similar to creating a new record except you'll also need to pass save() a JSON object that contains the updates you want to make:

new model({  
    // The query will match these parameters
    'id': 1
    // Will return row with ID 1
}).save({
    // These updates will be made
    'name': 'Joe'
    // Record's name will be updated to 'Joe'
}).function(updatedModel) { ... }).catch(function(err) { ... });

Fetch Record(s)

Fetching records also looks pretty similar to the previous operations except you will need to use fetch():

new model({  
    // Query params 
}).fetch().function(fetchedModel) {
    // Do stuff with fetchedModel 
}).catch(function(err) { ... });

Delete Record

Last but not least, you can delete a record using just about everything used in the fetch example while adding the destroy() function:

new model({  
    // Query params
}).fetch().then(function(fetchedModel) {
    fetchedModel.destroy(); 
}).catch(function(err) { ... });

You can avoid requiring the then() handling if your query includes the id of the row:

new model({  
    'id': 1
}).fetch().destroy();

Conclusion

This is as far as I have gotten with Bookshelf.js - I do intend to continue writing about this library as I learn more myself. I hope this helps somebody else getting started with Bookshelf.js! If you are interested in reading the official documentation you can find that here.

in python, space, virtualization, libvirt, development

When I first started working on Space I knew I would need to use something like libvirt to make all of the calls to the hypervisor its self. I had heard of libvirt several times during my tenure in the IT industry so I figured it would be realtively easy to get things running fairly quickly but when I started reading the libvirt documentation I was very surprised at how poor it was.

I'm not hoping to rewrite all of their documentation, but I would like to give a few examples for doing basic things using the libvirt API in Python.

Connecting to the Hypervisor

You'll need to establish a connection with your hypervisor before you can interact with it. A simple function to build that connection looks like this:

import libvirt

def connect():  
    connection = libvirt.open("SYSTEM_TYPE:///system")
    return conn

You'll end up returning a connection object that you can then use to do other things. SYSTEM_TYPE will be something like QEMU or XEN.

Getting a List of Domains (Virtual Machines)

There are actually two calls required to do this, listDefinedDOmains() which returns a list of all defined domains (running and not running), and listDomainsID() which returns a list of only running domains.

import libvirt

connection = libvirt.open("SYSTEM_TYPE:///system")

# This lists all domains on the host
connection.listDefinedDomains()  
# Returns
['vm1','vm2','vm3']

# This lists running domains on the host
connection.listDomainsID()  
# Returns
[2] 
# These are the ID's as used by the hypervisor - you'll need to store this somewhere to tie this to a virtual machine as it will change each time the virtual machine is started.

Find a Domain

You can find a domain two different ways, you can use the ID that was mentioned above (this changes, so it's tough to use this unless you maintain this information somewhere) or you can use the name of the virtual machine.

I generally use the name because I set it and it stays the same regardless of what happens to the virtual machine. The name is what you'll see if you run virsh list --all:

 virsh list --all
 Id    Name                           State
----------------------------------------------------
 2     vm54d42cd2a5ee223e6be95bc2     running

So you can do that using the following functions:

# Assume the connection function is included here

virtual_machine = connection.lookupByName("vm54d42cd2a5ee223e6be95bc2")  
# Returns the object representing this virtual machine

virtual_machine = connection.lookupByID(2)  
# Returns the same object, but uses the ID as shown in the virsh output. Probably not the one you want to use, but it is there if you need it.

Start Domain

Once you have the domain object you can start/stop/redfine it fairly easily. Let's look at starting it:

virtual_machine.create()  
# Will create (start) this virtual machine if it is defined
Stop Domain
virtual_machine.destroy()  
# Will destroy (stop) this virtual machine if it is defined

Create Domain

Now lets make a brand-spankin'-new domain. This wil take a bit of work, so lets dig in.

First you'll need to make a domain configuration file - this is an XML file that defines a bunch of things about the domain. I've provided an example below:

<domain type="kvm">  
    <name>vm11</name>
    <cpu>
        <topology cores="4" sockets="1" threads="4" />
    </cpu>
    <uuid>90e7bca4-97b2-11e4-86bf-001e682ee78a</uuid>
    <memory unit="MB">512</memory>
    <currentMemory unit="MB">512</currentMemory>
    <vcpu placement="static">2</vcpu>
    <os>
        <type>hvm</type>
        <boot dev="hd" />
    </os>
    <features>
        <acpi />
        <apic />
        <pae />
    </features>
    <clock offset="utc" />
    <on_poweroff>destroy</on_poweroff>
    <on_reboot>restart</on_reboot>
    <on_crash>restart</on_crash>
    <devices>
        <emulator>/usr/libexec/qemu-kvm</emulator>
        <disk device="disk" type="file">
            <driver cache="none" name="qemu" type="raw" />
            <source file="/var/disks/vm11.img" />
            <target dev="hda" />
            <address bus="0" controller="0" target="0" type="drive" unit="0" />
        </disk>
        <disk device="cdrom" type="file">
            <source file="/var/images/ubuntu-14.04.1-server-amd64.iso" />
            <driver name="qemu" type="raw" />
            <target bus="ide" dev="hdc" />
            <readyonly />
            <address bus="1" controller="0" target="0" type="drive" unit="0" />
        </disk>
        <interface type="network">
            <source network="default" />
        </interface>
        <graphics port="-1" type="vnc" />
    </devices>
</domain>  

I won't dig into how this is made (perhaps I'll do that in another post), but long story short you'll need to make this file and save it somewhere on your filesystem. You can take a look here to see how I did this in Space.

Next, we'll need to use this configuration to define a new virtual machine. Defining a virtual machine essentially means you are creating it, once it is defined you can undefine it (delete it), start it, and shut it down. To define the domain, you can use defineXML(xml):

# First you'll want to get your XML file into memory
xml = ""  
with open(path_to_xml_config, "r") as file:  
    xml = file.read()
connection.defineXML(xml)  
# If you don't get an error, the domain has been created, woot!

Delete Domain

Deleting a domain is significantly easier than making one. You'll just need to use the undefine() function:

virtual_machine = connection.lookupByName("your_domain_name")  
virtual_machine.undefine()  
# Bye bye virtual machine, its dead! 

Updating Domain

I'm sure some of you have put two and two together here and have pieced together how you would update a domain from that - if not, I'll detail that now.

You'll first need to overwrite the old configuration file (or make a new one). The new configuration file should include the new options you want to update the domain to. Next, you'll need to undefine the current domain using the method described above.

Once the domain has been undefined, simply define it again as described in the create a domain section. Start it up and you'll have a domain with updated settings.

Conclusion

I hope this saves somebody time in the future because it took me a while to figure out how to get all this stuff working my first time around. If you have any questions about anything I didn't cover, please feel free to comment or hit me up on Twitter @joseph_pettit.