Look alive. Here comes a buzzard.

Setting up automated backups for ESXi can be a little tricky. After quite a bit of searching I finally came up with these steps that worked for me. I don’t take credit for any of these. I’m just providing you with the steps that I followed and the posts that helped me.

DISCLAIMER: There are some unsupported steps here. Proceed at your own risk!

Step 1: Enable ssh access on your ESXi host. To do that, follow these steps (originally from this post).

NOTE: You actually don’t have to enable ssh for this, I just find it convenient to have remote access to the host machine.  If you don’t want to enable ssh, perform steps 1-3 below, then skip on to section 2.  When you reach section 3, rather than using ssh, you’ll just execute the commands directly on the host machine at the console.

1. From the console on your host (actually on the host machine - not using the Remote Client), press ALT-F1
2. Type in the text “unsupported” and press ENTER. Note: the text will not appear as you type.
3. Enter the root password
4. Enter the command

   vi /etc/inetd.conf

5. Find the line starting with “#ssh”. You can search by pressing / and then typing ssh and pressing Enter.
6. Once you’ve found that line, place the cursor on the # and press x. This should remove the # character.
7. Now type

   :wq!

   and press ENTER to save your changes.

8. Next you must restart the inetd process. To do that, you first need the process id. At the console, enter

   ps | grep inetd

   You should get something that looks like this:

   1273 1273 busybox      inetd

   In this case, 1273 is the process ID.  To kill the process enter kill -HUP <process id>.  So in this case, you would enter the following command:

   kill -HUP 1273

9. SSH is now enabled on your ESXi host machine. To test this you’ll need an ssh client. I’m using a mac so I can open terminal and enter the following command:

   ssh root@<myhost>

   If you’re using windows you may need something like PuTTY or Cygwin.

Step 2: Configure your backup datasource

Next you need to figure out where your backups will be stored.  It can be on the ESXi host if you like, but really it should be on a SAN or NAS or something like that.  I’m using a Netgear ReadyNAS NV+ and it works great.

1. Create a share where your backups will be stored and call it something like “backups”.  Make sure that anyone can read/write to this share.
2. Open the VMWare Infrastructure Client and connect to your host machine as root.
3. Go to the Configuration tab for the host and click Storage.
4. Click Add Storage… and create a new datastore pointing to the share you created in step 1.  Call this datastore something descriptive…  like “backups”

Step 3: Set up your backup script.

I found this great backup script.  It lets you specify the VMs you want to backup in a separate text file which makes it really easy to add new VMs to the rotation.  It also lets you keep a rotating number of VMs if you don’t want to over-write each backup daily.

1. Download the backup script.
2. Copy it to the same backup share you created in the last section.  You can create a directory called _scripts to keep it separate from the actual backups.
3. Open the script and customize the variables at the top.  In my case, all I had to modify was the VM_BACKUP_VOLUME variable.  If you named your datastore “backups” you can edit this line to look like this:

   VM_BACKUP_VOLUME=/vmfs/volumes/backups

    

4. Create a new text file in the same directory as the backup script called VMs.txt.  In this file enter the name of each VM you want to back up.  Each VM name should be on a new line.  Your file should look something like this:

   VM1
   VM2
   A_THIRD_VM

5. Connect to your host using ssh i.e. ssh root@<myhost>
6. Enter this command:

   vi /var/spool/cron/crontabs/root

7. Move the cursor to the very end of the file. Press a. This will enter edit mode.
8. Press Enter to create a new line and then add the following text:

   02 02 * * *  sh /vmfs/volumes/backups/_scripts/ghettoVCB.sh /vmfs/volumes/backups/_scripts/VMs.txt

   This will run the backup script at 2:02 AM every day. You can change the frequency or time of day if you like. This crontab reference is helpful.

9. At this point your entire file should look something like this:

    #syntax : minute hour day month dayofweek command
   01 01 * * * /sbin/tmpwatch.sh
   01 * * * * /sbin/auto-backup.sh #first minute of every hour (run every hour)
   00,10,20,30,40,50 * * * * /sbin/decodeSel.sh #Every 10 minutes, translate the latest IPMI SEL data
   02 02 * * *  sh /vmfs/volumes/backups/_scripts/ghettoVCB.sh /vmfs/volumes/backups/_scripts/VMs.txt
   

   If you are satisfied, press Escape.

10. Type the following command to save the file and exit::wq!
11. The backups are now scheduled

At this point, you now have backups scheduled to run every day at 2:02 AM. You can change which VMs are backed-up by editing the VMs.txt file you created. You could even set up multiple cron jobs with different backup frequencies, and different VMs.txt files to back up different VMs at different times.

Before finishing however, you might want to do a test run. If you are still connected to the host machine using ssh, enter the following command:

sh /vmfs/volumes/backups/_scripts/ghettoVCB.sh /vmfs/volumes/backups/_scripts/VMs.txt

IMPORTANT: This will take a while if you have large VMs. You may want to edit VMs.txt to target a single small VM as a test run. If it works for that, it should work for the rest.

One last thing. Unfortunately, the crontabs/root file isn’t saved after a reboot. In order to avoid adding the scheduled job after each reboot you may have to follow one of the techniques described here.

It struck me the other day that the way I use source control is very similar to how I play video games - specifically, when and how I save my game.  I like to use source control as a safety net - it’s a way to keep track of a place I may need to return to.  It allows me to continue on with my work and not be worried if I go down a wrong path.  I can always revert and be right back where I started.

Taking this one step further, we might say that the quick save in a video game is like a commit in source control. Saving your game with a specific name is like creating a branch.

Let’s say I just finish taking out a whole horde of zombies (Left 4 Dead anticipation is infiltrating my blog).  I’ll probably do a quick-save to make sure I don’t have to re-play the same encounter again.  This is just like committing some code you just finished.  You’re saving your current progress to make sure you don’t have to re-write the same code twice.

Now, back to my game, what if I come to a fork in the road.  I can go left through the ominous looking sewer, or I can go right through the abandoned city.  Well at this point I’ll go and save my game with some sort of name like “Taking the sewer”.  That way I can continue playing, quick-saving as I go, but I can always get back to that fork if it turns out I made a mistake.

This is very similar to creating a branch in source control. You want to start working on something new, or something experimental, and you need to be able to save your progress as you go, but you always may need to go back to that original point.

I find that many developers wait too long to commit their changes.  The more frequently you commit, and the more organized you are about it, the less you have to worry about keeping track of every change you make.  This means you have left cruft floating around in your mind to remember and you can concentrate on writing good code.  I think if developers think about source control the same way they think about saving their game progress, it would help them get into the habit of committing frequently and branching appropriately.

I wanted to do a quick post on why I chose Ruby for RPoint. I talked about this a little in the first post but just looking at the examples I’ve given so far, one could make the argument that rather than creating these scripts in ruby, you could just create a simple XML document that defines your structure, templates, etc.

This is very true - and in fact I have written tools like this in the past. But here’s the problem I’ve encountered with XML. You end up trying to build a programming language out of XML. That’s basically where CAML has ended up. And I don’t know a single person who actually enjoys looking at, much less creating, CAML.

Here’s what I like about the ruby approach. It let’s you write something like this:

site_names = ["Site One", "Site Two", "Site Three"]
at "http://localhost" do
  site_names.each do |name|
    create_web name, TeamSite
  end
end

Just the simple addition of an array of site names is something that would be hard to do in XML. What if you wanted to pull your information from a database? Or a file system? Or an Excel spreadsheet? For any of those you would have to write more logic into your XML mini-language - constructs like looping, conditionals, some mechanism to handle plug-ins for additional logic, etc. By leveraging an actual programming language, we get all of that for free.

Want to create 50 test sites? Here you go.

at "http://localhost" do
  50.times do |i|
    create_web "Site #{i}", BlankSite
  end
end

’nuff said.

Yesterday I posted about RPoint for the first time and I showed you a simple script to create a site structure. All of that code is working today. I wanted to keep the example simple and show you something that was actually functional.

Today I want to dream a little. Here are a few scripts I want RPoint to be able to support.

Creating a List Template

This script would create a list template.

class Milestones
  include ListTemplateMixin

  name "Milestones"
  description "A Simple Milestone List"

  column "Milestone", Text, :is_title => true, :required => true
  column "Description", RichText
  column "Date", Date, :required => true, :default => :today

  view "All Milestones" do
    show_columns "Milestone", "Date"
  end

  view "Overdue Milestones" do
    show_columns "Milestone", "Date"
    where :date => less_than(:today)
    order_by :date => :descending
  end
end

Once you had defined your template, you could use it in a few different ways. First, you could use it in a site creation script:

at "http://localhost" do
  create_list "My Milestones", Milestones
end

Second, and possibly cooler, you could generate a list definition:

>ir generate.rb list_definition --source MilestoneTemplate.rb

Migrations

Migrations in Rails is a great pre-defined structure for handling your database schema and changes to it over time. It may not be anything too ground-breaking on its own, but it enforces a discipline when it comes to managing your database. You are encouraged to think about situations where you want to change your database but data already exists. You can write scripts to migrate existing data to your new structure and it’s all managed within source control as an ordered list of steps to execute.

I want to be able to do something similar with RPoint.

# first migration - create my site collection
class CreateSiteMigration
  at :webapp => "http://localhost" do
    create_site "My Site", TeamSite
  end
end

# second migration - add a list and add a lookup to that list on an existing list
class AddIssuesMigration
  at "http://localhost/sites/mysite" do
    create_list "Issues", Issues
    update_list "Tasks" do
      add_column "Related Issue", Lookup("Issues", :field => :title)
    end
  end
end

Okay - that’s enough dreaming for now. Just incase anyone missed the intro THIS CODE DOES NOT WORK. It is completely imaginary. I am hoping that something along these lines will become reality. And I hope that you will agree that this would be a much nicer way to create SharePoint solutions than slogging through hundreds of lines of XML.

A few days back I talked about how the release of IronRuby would be great for SharePoint scripting. In my opinion, what makes Ruby such a powerful language is that it is great for creating domain specific languages (DSLs). I’m going to keep this post short so I won’t go into what DSLs are or examples of them. If you want to find out more info, look at some other ruby libraries such as rspec, rake, and datamapper.

I think that SharePoint is in dire need of better tools to facilitate solution development, deployment, and maintenance. It is simply too difficult to build a site in one environment and then deploy it in production. Sure you can do backup and restore, but what about when you want to make some upgrades?

Anyway, this weekend I started working on an IronRuby-based DSL for SharePoint scripting. I’m calling it RPoint for now (until I think of something sexier!). Now, this is very very very early in development and I am sure things aren’t set up today as nicely as they should be long term, but I’ve been really excited by the progress and the possibilities.

RPoint will allow you to write code like this to create SharePoint sites:

# create a new subweb beneath an existing site
at "http://localhost" do
  create_web "New Web", TeamSite
end

# create a new site collection and website structure
at :webapp => "http://localhost" do
  create_site "New Site Collection", BlankSite, "domain\\user" do
    inside root_web do

      # create our site structure
      create_web "Child Web", TeamSite
      create_web "Another Child", BlankSite do
        create_web "Private Site", TeamSite, :inherit_permissions => false
      end

      # create a few lists
      create_list "Documents", DocumentLibrary
      create_list "Announcements", Announcements

    end
  end
end

I think that’s pretty cool. Beyond simply creating sites, RPoint will allow you to create lists, views and probably upload resources like images and documents.

My goals for RPoint are to be able to do the following:

• Easily write scripts to create site structures and lists
• Rails-like “Migrations” for applying changes to existing sites
• Custom site template and list definitions in ruby as opposed to XML
• Ability to run template/list definitions against a live site or generate schema.xml/onet.xml files for production deployment
• Support web-services automation as well as object model automation (this will be tough)

SharePoint is a huge product. The code-base is enormous and the aspirations for RPoint are lofty. I would greatly appreciate any assistance. The more hands there are to help, the sooner we will have a tool to make SharePoint development a little easier.

To get RPoint in its current form, you can download it from GitHub.

I wrote a while back about SharePoint scripting with IronPython. Well today’s announcement of the first binary release of IronRuby got me pretty darn excited. I really like Python, but I LOVE Ruby. I’ve been waiting for IronRuby to catch up to IronPython for quite a while and it looks like they’re getting close.

Here’s a quick guide to getting up and running with IronRuby and SharePoint.

Installation:

1. First, download the IronRuby release from Rubyforge.
2. Next, copy the ironruby folder you extract to your Program Files (put it wherever you like - that’s where I put mine).
3. Update your system’s PATH environment variable to contain the path to the ironruby\bin folder.  So if you placed it in your Program Files, you would add this to your PATH: c:\program files\ironruby\bin

Creating a Script:

Next you’ll want to create a new text file called print_sites.rb. I recommend the e text editor by the way - great tool.  Add the following lines to your file:

require 'System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089, processorArchitecture=MSIL'
require 'Microsoft.SharePoint, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c, processorArchitecture=MSIL'

Uri = System::Uri
SPSite = Microsoft::SharePoint::SPSite
SPWebApplication = Microsoft::SharePoint::Administration::SPWebApplication

Okay - what does that do? Well the first two lines are like adding a reference to a standard Visual Studio project. They tell IronRuby that we need to reference (require) the System and Microsoft.SharePoint assemblies. The next three lines are for convenience. They set up aliases to the System.Uri class and two SharePoint classes. In our ruby code, now we can just use Uri, SPSite, etc instead of having to type out the fully qualified path.

Next, let’s add some ruby code to enumerate all sites in a web application.

webapp = SPWebApplication.Lookup(Uri.new("http://localhost"))
webapp.Sites.each do |site|
  puts site.Url
end

(hmm, looks like I don’t actually need the SPSite alias after all)

Now, save your file. You can execute it like this:

> ir print_sites.rb

If everything is set up right, you should see the URL for each site collection be printed to the screen.

There you go. Have fun.

I wrote a small introduction to SharePoint scripting with Python a while back and this weekend I didn’t have anything better to do (well actually I did, but I’m an addict - what can I say) so I decided to expand on it.  I’ve started putting together a small Python library for working with SharePoint using IronPython.  It’s called - amazingly - sp.py.

So far the library if pretty small - although it does include a nice little module for workign with stsadm commands.  I’ll be adding to it as necessary.  I’ve decided that although adding custom stsadm extensions is nice and does fit into the “way to do things in SharePoint”, it’s just too much overhead when you want to do something simple.  So at this point when things come up that need automation, I’ll be adding new sp.py scripts.

In addition to the sp.py library I’ve added three scripts:

• applytheme.py - apply a theme to all webs in a web application
• backupsites.py - backup all site collections in a web application
• setprofileprop.py - set a property on all user profiles based on a dynamic pattern

So where can you get this awesomeness?  Well, actually you can get it on github.  Github???  You mean that totally web 2.0 site for hosting Git repositories?  Yeah - that’s the place.  If you’re wondering why I’m using git for source control, well I guess Subversion just wasn’t “edge” enough anymore.  Git’s actually pretty cool - and if you need to use Windows you can download a Windows version here.

At some point I’ll figure out how to post it as a zip or something instead of a tarball.  But until that point, you can grab the tarball here.

As a long time Microsoft developer who is in the progress of branching out and learning new things, I sometimes find it hard to give up certain features I’ve grown accustom to.  When it comes to writing code, one feature of Visual Studio I have learned to rely on is folding sections of code using #region and #endregion.  I find regions to be a great way to organize code around different subjects, activities, etc.  For me it’s a way to reduce the visual clutter and immediately jump to the section of code I need to work on.

On the Mac, TextMate does a great job of code folding based on syntax.  I’m doing a fair amount of work with Ruby these days and TextMate is an excellent tool.  Unfortunately for me however all folding is based on the syntax of the code, and not extra non-code markers like #region.  Luckily TextMate’s bundle system is completely extensible and customizable.  Folding is based on regular expressions so with a fairly simple change I was able to get #region folding in TextMate.

The first step is to figure out the regular expression.  The core regex for #region folding would be this:

^\s*\#region
^\s*\#endregion

Next open up TextMate’s Bundle editor, open the Ruby language, and locate  the foldingStartMarker and foldingStopMarker sections.  Below is the entire contents of my foldingStartMarker and foldingStopMarker sections.

foldingStartMarker = '(?x)^
(\s*+
(module|class|def
|unless|if
|case
|begin
|for|while|until
|(  "(\\.|[^"])*+”          # eat a double quoted string
| ”(\\.|[^''])*+”        # eat a single quoted string
|   [^#"'']                # eat all but comments and strings
)*
(                 \s   (do|begin|case)
| [-+=&|*/~%^<>~] \s*+ (if|unless)
)
)\b
(?! [^;]*+ ; .*? \bend\b )
|(  “(\\.|[^"])*+”              # eat a double quoted string
| ”(\\.|[^''])*+”            # eat a single quoted string
|   [^#"'']                    # eat all but comments and strings
)*
( \{ (?!  [^}]*+ \} )
| \[ (?! [^\]]*+ \] )
)
).*$
|   [#] .*? \(fold\) \s*+ $         # Sune’s special marker
|    ^\s*\#region
‘;
foldingStopMarker = ‘(?x)
(   (^|;) \s*+ end   \s*+ ([#].*)? $
|   ^     \s*+ [}\]] \s*+ ([#].*)? $
|   [#] .*? \(end\) \s*+ $    # Sune’s special marker
|   ^\s*\#endregion
)’;

You can see where my two regular expressions were added near the end of each section.

Once you do that, close the editor, reload your bundles, and you’re good to go.

I’ve had this project in the works for a while now and it’s time to open it up to the world. The SharePoint Knowledge Base is a simple wiki site I created to capture some best practices, how-tos, tips and tricks, etc. While I think that blogs are great tools to communicate this sort of information, I’m not satisfied with the experience of searching for a specific issue or problem. Often times I find myself sifting through numerous blog posts - sometimes with conflicting information - left wondering which author knows his or her stuff.

The intent of the SharePoint Knowledge Base is to provide a wikipedia-like site dedicated to capturing community-driven expert content around SharePoint.

Today, there is not a lot of content. That’s where you come in. We all come across things every day that the rest of the community can benefit from. Why not take a minute and jot down your notes here in the SharePoint KB? Not only for your own future reference, but for the benefit of others as well.

There is no registration required to create or edit content. Until we start to have problems this site will remain completely open for anyone to edit.

So please, if you have something to contribute I would encourage you to post it here.  We can all benefit from a better way to store and update SharePoint-related information.  There’s so much of it out there but it’s so hard to sift through.  I hope that together we can start to chip away at this problem.

I just came across this amazing resource of field types and their XML schema.

 http://www.johnholliday.net/download/fieldswss.htm