Or: How to covert Microsoft’s Internet Explorer Application Compatibility Virtual PC images for testing on a Mac with Parallels Desktop or VMware Fusion. (but that’s kind of a mouthful)

To help with web development and testing, Microsoft provides several compatibility environments for past versions of Internet Explorer. These are time-limited, pre-configured Virtual PC disk images, intended for use with Windows Virtual PC.

That’s great, except Virtual PC doesn’t really run inside another virtual environment like VMware Fusion or Parallels Desktop. Since so much web development is done on Macs these days, this makes testing difficult. But, the VPC images are just virtual hard drives, so they can be easily converted to the native format of either Parallels or VMware.

Below are instructions for converting Microsoft’s .VHD files to native Parallels Desktop or VMware Fusion virtual machines. The goal is to do everything without using another Windows machine or digging out your old WinXP CD.

Get the IE Compatibility VMs

Download one or all of the Internet Explorer Application Compatibility VPC Images. Self-extracting EXEs can be decompressed on a Mac with The Unarchiver. The VHD is all that’s needed, one or all of these:

• IE6Compat.vhd
• IE7Compat.vhd
• IE8Compat.vhd
• Vista Business with IE7.vhd
• Vista SP1 x86 with IE8 2009-Apr.vhd

Convert the VHD image to Parallels’ or VMware’s native virtual disk format

Currently, Parallels Desktop 6 can’t convert VHDs through it’s GUI, but it does include a hidden command-line utility which works perfectly. Create a new Parallels VPC in the default location (~/Documents/Parallels) with this command:

$ /Library/Parallels/Parallels\ Service.app/Contents/PlugIns/Parallels\ VM\ Converter.app/Contents/MacOS/prl_convert IE7Compat.vhd 

For VMware Fusion, use one of the components in Q (qemu-img, details) to convert the .VHD files to VMware’s .vmdk format.

$ /Applications/Q.app/Contents/MacOS/qemu-img convert -O vmdk -f vpc IE7Compat.vhd IE7Compat.vmdk 

Create a new Virtual Machine using the vmdk as the disk. I could not get Vista to work dependably with VMware.

XP: Fix Windows Drivers

On first boot, the VM will attempt to re-activate Windows. This will fail because the networking drivers haven’t been installed yet.

Next the VPC_EULA will open and the first of many “Files Needed” dialogs and “Found New Hardware” wizards will appear. Close and cancel everything until you’re back at the Windows XP desktop.

If you’re running Parallels Desktop, it will take over and install its tools. If you’re using VMware, choose Install VMware Tools from the Virtual Machine menu. VMware also asks for a handful of drivers during tools installation, cancel these, they’ll be fixed later. The VM will reboot when the tools and drivers finish installing.

After the initial reboot, Windows requires WGA activation. Since this is a Microsoft-provided license, the VM will pass WGA and Windows will be activated. There’s no point in registering, so skip that. Everything from here forward is cleanup.

The easiest way to fix a bunch of drivers is to install XP Service Pack 3. Download the ISO from Microsoft here: Windows XP Service Pack 3 – ISO-9660 CD Image File

Any lingering driver issues can be resolved in Device Manager (Control Panel > System > Hardware tab > Device manager). The problem items should either have their drivers updated or, if that doesn’t work, disabled altogether. VMware’s audio driver failed to load initially but updated successfully. Mystery USB devices were a problem in both, but can be disabled.

Ironically, I had much more trouble getting these VMs running natively under Virtual PCs on a Windows box. Many of these steps are necessary there too.

Other stuff I install:

• Microsoft Script Debugger
• Bonjour for Windows

VMware or Parallels?

For years I’ve been bouncing back and forth between the two. It seems that with each major release, one leapfrogs the other. After testing VMs in both, I’ve decided to switch back to Parallels. At the moment, it just seems more full-featured and seamless. According to a recent MacTech virtualization comparison test, Parallels is also faster.

Disclaimer

The above process may violate the second paragraph of the “Installation and Use Rights” section of Microsoft’s XP EULA:

You may install and use one copy of the software on your device of which you are running a validly licensed copy of Microsoft Virtual PC or Microsoft Virtual Server. You may not change or convert the virtual hard disk image from the VHD format.

However, no such clause appears in the Vista EULA which seems to contradict the above and fully supports virtualization:

You may install and use one copy of the software on your device of which you are running any virtualization software able to run Virtual Hard Disk (VHD) formats, including Microsoft Virtual PC or Microsoft Virtual Server.

Anyway, I assume no responsibility for what anyone does with this information.

Fact is, to get Gitweb working only one line needs changing. After the following edit, all local configuration values can be loaded from a simple config file.

On line 546, insert the name of your config file:

-our $GITWEB_CONFIG = $ENV{'GITWEB_CONFIG'} || "++GITWEB_CONFIG++";
+our $GITWEB_CONFIG = $ENV{'GITWEB_CONFIG'} || "gitweb_config.perl";

A set of fully-documented configuration files is available in the Simple Gitweb Config project on Github, to help get things up and running quickly.

Full instructions

Here are all the steps necessary to get Gitweb running on a webserver. In most cases, the configuration will work without modification, installation shouldn’t take more than five minutes start to finish.

1. Create temporary source directory:

mkdir ~/src && cd ~/src

2. Download and extract Git source and Simple Gitweb Config tarballs:

curl -LO http://kernel.org/pub/software/scm/git/git-1.6.6.tar.gz
curl -LO http://github.com/joemaller/Simple-Gitweb-Config/tarball/master
tar -xvzf git-1.6.6.tar.gz
tar -xzvf master

3. Copy Git’s gitweb subdirectory to a web accessible location, then copy the gitweb config files into the live gitweb directory:

cp -r git-1.6.6/gitweb ~/www/
cp joemaller-Simple-Gitweb-Config*/* ~/www/gitweb/

4. Replace ++GITWEB_CONFIG++ with gitweb_config.perl and make the resulting CGI executable:

sed -re's/\++GITWEB_CONFIG\++/gitweb_config.perl/g' gitweb.perl  > gitweb.cgi
chmod a+x gitweb.cgi

5. Gitweb should be working now, but if necessary, edit projects_list.txt, gitweb_config.perl and customize the three html files for your environment.

One last note, since it wasn’t immediately obvious to me. To change the default repository description, “Unnamed repository; edit this file to name it for gitweb.” edit the configuration file inside the .git folder. Descriptions are per-repository and do not update or transfer with pushes, pulls or clones.

Those little search bubbles that popup on the New York Times website whenever you select text really annoy me. Clicking this bookmarklet on any NYTimes page will prevent them from appearing:

NYTfix

This is a 2-minute solution, there’s no domain checking or anything, all it does is remove existing bubbles then cancel the document’s mouseup observer, which the NYTimes site uses to trigger the search balloons. The bookmarklet was very quickly checked in Safari, Firefox and IE8, NYTimes text selection doesn’t work at all for me in IE6 or IE7.

Information and schedules:

• District of Columbia Inauguration Day 2009
• Presidential Inaugural Committee

The official schedule has the swearing in ceremony as starting at 11:30 am. I only found one source listing the Vice-President’s swearing in at 11:45 and President’s at noon.

Live video streams

The following sites have announced live streaming coverage of the inauguration.

Video sites:

• Hulu
• Joost
• UStream (supposedly iPhone compatible)
• YouTube (maybe)

News:

• ABC News
• CBS News
• CNN (partnered with Facebook)
• MSNBC
• Washington Post
• New York Times

Governmental sites:

• C-SPAN
• Presidential Inaugural Committee
• US Congress

Other “where to watch” articles:

CNN, CNet, PC World, WebTVhub, TechCrunch

I won’t be at all surprised if the Internet melts on Tuesday, just before noon EST. But I hope it doesn’t.

Some key advantages:

• Pushing remote changes automatically updates the live site
• Server-based site edits won’t break history
• Simple, no special commit rules or requirements
• Works with existing sites, no need to redeploy or move files

Overview

The key idea in this system is that the web site exists on the server as a pair of repositories; a bare repository alongside a conventional repository containing the live site. Two simple Git hooks link the pair, automatically pushing and pulling changes between them.

The two repositories:

• Hub is a bare repository. All other repositories will be cloned from this.
• Prime is a standard repository, the live web site is served from its working directory.

Using the pair of repositories is simple and flexible. Remote clones with ssh-access can update the live site with a simple git push to Hub. Any files edited directly on the server are instantly mirrored into Hub upon commit. The whole thing pretty much just works — whichever way it’s used.

Getting ready

Obviously Git is required on the server and any local machines. My shared web host doesn’t offer Git, but it’s easy enough to install Git yourself.

If this is the first time running Git on your webserver, remember to setup your global configuration info. I set a different Git user.name to help distinguish server-based changes in project history.

$ git config --global user.name "Joe, working on the server"

Getting started

The first step is to initialize a new Git repository in the live web site directory on the server, then to add and commit all the site’s files. This is the Prime repository and working copy. Even if history exists in other places, the contents of the live site will be the baseline onto which all other work is merged.

$ cd ~/www
$ git init
$ git add .
$ git commit -m"initial import of pre-existing web files"

Initializing in place also means there is no downtime or need to re-deploy the site, Git just builds a repository around everything that’s already there.

With the live site now safely in Git, create a bare repository outside the web directory, this is Hub.

$ cd; mkdir site_hub.git; cd site_hub.git
$ git --bare init
Initialized empty Git repository in /home/joe/site_hub.git

Then, from inside Prime’s working directory, add Hub as a remote and push Prime’s master branch:

$ cd ~/www
$ git remote add hub ~/site_hub.git
$ git remote show hub
* remote hub
  URL: /home/joe/site_hub.git
$ git push hub master

Hooks

Two simple Git hooks scripts keep Hub and Prime linked together.

An oft-repeated rule of Git is to never push into a repository that has a work tree attached to it. I tried it, and things do get weird fast. The hub repository exists for this reason. Instead of pushing changes to Prime from Hub, which wouldn’t affect the working copy anyway, Hub uses a hook script which tells Prime to pull changes from Hub.

post-update – Hub repository

This hook is called when Hub receives an update. The script changes directories to the Prime repository working copy then runs a pull from Prime. Pushing changes doesn’t update a repository’s working copy, so it’s necessary to execute this from inside the working copy itself.

#!/bin/sh

echo
echo "**** Pulling changes into Prime [Hub's post-update hook]"
echo

cd $HOME/www || exit
unset GIT_DIR
git pull hub master

exec git-update-server-info

post-commit – Prime repository

This hook is called after every commit to send the newly commited changes back up to Hub. Ideally, it’s not common to make changes live on the server, but automating this makes sure site history won’t diverge and create conflicts.

#!/bin/sh

echo
echo "**** pushing changes to Hub [Prime's post-commit hook]"
echo

git push hub

With this hook in place, all changes made to Prime’s master branch are immediately available from Hub. Other branches will also be cloned, but won’t affect the site. Because all remote repository access is via SSH urls, only users with shell access to the web server will be able to push and trigger a site update.

Conflicts

This repository-hook arrangement makes it very difficult to accidentally break the live site. Since every commit to Prime is automatically pushed to Hub, all conflicts will be immediately visible to the clones when pushing an update.

However there are a few situations where Prime can diverge from Hub which will require additional steps to fix. If an uncommitted edit leaves Prime in a dirty state, Hub’s post-update pull will fail with an “Entry ‘foo’ not uptodate. Cannot merge.” warning. Committing changes will clean up Prime’s working directory, and the post-update hook will then merge the un-pulled changes.

If a conflict occurs where changes to Prime can’t be merged with Hub, I’ve found the best solution is to push the current state of Prime to a new branch on Hub. The following command, issued from inside Prime, will create a remote “fixme” branch based on the current contents of Prime:

$ git push hub master:refs/heads/fixme

Once that’s in Hub, any remote clone can pull down the new branch and resolve the merge. Trying to resolve a conflict on the server would almost certainly break the site due to Git’s conflict markers.

Housekeeping

Prime’s .git folder is at the root level of the web site, and is probably publicly accessible. To protect the folder and prevent unwanted clones of the repository, add the following to your top-level .htaccess file to forbid web access:

# deny access to the top-level git repository:
RewriteEngine On
RewriteRule \.git - [F,L]

Troubleshooting

If you’re seeing this error when trying to push to a server repository:

git-receive-pack: command not found
fatal: The remote end hung up unexpectedly

Add export PATH=${PATH}:~/bin to your .bashrc file on the server. Thanks to Robert for finding and posting the fix.

Links

These didn’t fit in anywhere else:

• Toolman Tim has a very good introductory walkthrough of setting up a new remote git repository.
• This ended up being somewhat similar to the update mechanism in Ikiwiki, wish I’d found that page earlier.
• Getting a static web site organized with git came up with a different solution, but calling git reset --hard from a hook on the server could cause a lot of trouble when committing server-side changes.

Installing Git on a shared hosting account is simple, the installation is fast and like most things Git, it just works.

As with my previous Subversion on shared hosting post, this will be a barebones install. The purpose of this installation is to be able to push changes from remote repositories into the hosted repo, where the hosted repository may also serve as the source directory of the live website. Like this.

Prerequisites

The only two things you absolutely must have are shell access to the account and permission to use GCC on the server. Check both with the following command:

$ ssh joe@webserver 'gcc --version'
gcc (GCC) 4.1.2 20080704 (Red Hat 4.1.2-50)
[...]

If GCC replies with a version number, you should be able to install Git. SSH into your server and let’s get started!

If you see something like /usr/bin/gcc: Permission denied you don’t have access to the GCC compiler and won’t be able to build the Git binaries from source. Find another hosting company.

Update your $PATH

None of this will work if you don’t update the $PATH environment variable. In most cases, this is set in .bashrc. Using .bashrc instead of .bash_profile updates $PATH for interactive and non-interactive sessions–which is necessary for remote Git commands. Edit .bashrc and add the following line:

export PATH=$HOME/opt/bin:$PATH

Be sure ‘~/opt/bin’ is at the beginning since $PATH is searched from left to right; to execute local binaries first, their location has to appear first. Depending on your server’s configuration there could be a lot of other stuff in there, including duplicates.

Double-check this by sourcing the file and echoing $PATH:

$ source ~/.bashrc
$ echo $PATH
/home/joe/opt/bin:/usr/local/bin:/bin:/usr/bin

Verify that the remote path was updated by sending a remote command like this (from another connection):

$ ssh joe@webserver 'echo $PATH'
/home/joe/opt/bin:/usr/local/bin:/bin:/usr/bin

Note: Installing into the ~/opt directory keeps the home folder cleaner and is where add-on applications are customarily installed on Unix systems.

Installing Git

SSH into your webserver. I created a source directory to hold the files and make cleanup easier:

$ cd
$ mkdir src
$ cd src

Grab the most current source tarballs from the Git site. At the time this post was last updated, the most recent version was v1.7.6:

$ curl -LO http://kernel.org/pub/software/scm/git/git-1.7.6.tar.bz2

Untar the archive and cd into the new directory:

$ tar -xjvf git-1.7.6.tar.bz2
$ cd git-1.7.6

This next step is the only one that really seems to matter with regards to shared hosting. The Configure script needs to be told where to install, and because we’re on a shared host, Git’s files should be put somewhere in our home directory:

$ ./configure --prefix=$HOME/opt
[words...]

Lastly, make and install:

$ make && make install
[lots of words...]

That should be it, check your installed version like this:

$ git --version
git version 1.7.6

It's now safe to delete the src folder which contained the download and source files.

Note that these instructions do not install Git's documentation man pages. Also, these instructions appear to work exactly the same on Mac OS X, though the installer package is way easier and includes documentation.

My preferred shared hosting providers are A2 Hosting and WebFaction.

• YouTube.com pages are now supported (see notes)
• Daily Motion videos are supported for new-style urls (see notes)
• Kink.fm player and listings page are now supported
• Sideload.com play links are now supported
• WordPress Blogs using Viper Video QuickTags are supported for YouTube
• All media links now open into new windows, so you won’t have to re-transmogrify a page with several media files after playing one. Note that this is dependent on the iPhone, sometimes it will blank other windows)
• Some content in iframes will now be converted.
• MotionBox, Viddler and Vimeo embedded videos, while not supporting iPod/iPhone alternate content, now link to their respective detail pages.

The main bookmarklet code was updated. This was necessary to workaround a frustrating oversight with Google Code hosting. Everyone will need to update their bookmarklet, in the future all updates will be automatic.

This has turned out to be far bigger than I ever imagined. Thank you to everyone for the links, feedback, compliments and ideas.

Known issues

LiveJournal pages redefine a bunch of core JavaScript functionality, breaking all kinds of stuff including jQuery. Additionally, they’re serving media in an iframe from a different domain, meaning JavaScript couldn’t access the frame even if they hadn’t broken it.

Notes

YouTube Internal pages
Because of a strange iPhone quirk, these links all need to go through the Google redirector, otherwise they bounce back to uk.youtube.com instead of playing.

DailyMotion
DailyMotion videos using new-style urls, which are usually about six digits long, work correctly. Videos using the old-style alphanumeric ID do not work yet. I’m probably just going to resort to building a simple web-service to grab those. Additionally, there is no way to programatically access the mp4 alternate content url, so I just linked to their iPhone pages. I’d prefer embedding QuickTime directly, but it’s just not possible yet.