Migrating from TFS to Git

Sometimes the existing tools are making it difficult to do the things you need to do. In our case we needed a way to do multiple things at the same time:

  • Implement and test features in isolation
  • Fix things in production
  • Have code reviews
  • Have continuous integration for all branches
  • Test and review the next release(s)

TFS’s way of having physical folder per branch and merges happening on server side wasn’t just working for us and it reminded me of the following CommitStrip.

Git, SVN or...?

So this fall I migrated the team from TFS to Git. The actual migration included the following:

  • Migrating source code (4 TFS projects)
  • Setting up continuous integration (TeamCity)
  • Setting up automated deployments (Octopus, not done by me)
  • Short training and instructions

The source code was already hosted in Visual Studio Team Services (VSTS) so there was no need to migrate other project related data such as work items.

Plan

Before jumping in I wrote down all the things I needed to do in order to successfully implement the migration:

  • Write short instructions on how to use basic Git features (Git was new to most people)
  • Test the migration, continuous integration and automated deployments before the Big Day
  • Give training on Git (basic usage from the command line and Visual Studio)
  • Make sure everyone can clone test repository, create pull requests etc.
  • Configure server side (access rights, policies, branches, pull requests)
  • Migrate every branch from every TFS project with complete version history
  • Support Git usage via Skype/face-to-face

When it comes to actual development model I decided to go with a successful Git branching model with the addition of using pull requests. It is a proven model and quite simple. Developers do most of the work in feature branches and merge them into main development branch. Every now and then fixes must be merged into multiple branches but that is about as complicated as it gets.

Tools

As the plan was to migrate full version history I needed a tool to do the heavy lifting. Google gave me Git-TF and git-tfs. The Git-TF project hadn’t been active for a while and it seems it is dead.

Unfortunately, we do not have any distinct plans regarding the new official git-tf release. Sorry for inconvenience.

This left me with only one choice. Luckily the git-tfs project was actively developed and it looked to have the features I would need.

Testing

As I started test migration I noticed that git-tfs wasn’t able to traverse the branch trees without problems. Every now and then it would stop and ask me to manually input the changeset id:

warning: git-tfs was unable to find the root changeset (ie the last common commit) between
the branch '$/Project/QA'
and its parent branch '$/Project/Production'.
(Any help to add support of this special case is welcomed!
Open an issue on https://github.com/git-tfs/git-tfs/issues )
To be able to continue to fetch the changesets from Tfs,
please enter the root changeset id between the branch '$/Project/QA'
and its parent branch '$/Project/Production' by analysing the Tfs history...
Please specify the root changeset id (or 'exit' to stop the process):

Finding the correct changeset between two branches isn’t straight forward. If you write the wrong id, then the end result of the migration is wrong for some branches. After running into this issue more than few times I started to look into the branches. I found out that there have been lot of merges between different branches. It wasn’t just from develop to master and back.

Theory vs. Reality

The migration would sometimes go into infinite loop because of all the strange merges back and forth that had been done. At this point I decided that I would migrate the whole version history from the main development branch and manually take the latest changes from the other important branches. After all the other branches didn’t contain changes that were not in the main development branch.

Continuous Integration and Deployment

We were already using TeamCity and we decided to continue to do so. Setting up the builds was easy. Even if you are not familiar with TeamCity all you have to do is to create project, configure the version control connection (tell TeamCity address of Git repository) and setup build steps. Automated deployment was done with Octopus.

Instructions and Training

I didn’t want to end up in a situation where developers are asking me why they cannot clone the repository or how to do branches etc. so I did the following:

  1. Write instructions on how to install Git, configure it etc.
  2. Write short tutorial how to do daily things (branches, pull requests etc.)
  3. Create test repository and let people try themselves

When the decision to start using Git was made, I already knew I would be giving training using command line. No GUIs. Just written commands. But as time passed and I was solving other issues I started to second guess my decision. I came into the conclusion that using only command line in the training would look like the following xkcd comic:

Git

The team had a lot of people who had no experience on Git. Most had used only version control via GUI. What would be the point of showing command line usage in one-hour training? After all the goal was to have everyone working after the migration. So I changed my mind and I used Visual Studio’s Git integration because it had the features developers use every day. There is plenty of time to learn neat command line tricks and Git philosopfy later.

Migration

Note that creating 1:1 copy of your TFS project might take a long time for large projects. I started the migration by exporting the main development branch into Git repository. This will fetch every changeset from TFS and create equal commit history into the Git repository.

git tfs clone https://company.visualstudio.com:443 $/TFSProject/BranchName . --branches=none --export

If you do this in Windows you might run into issue where the path to repository is changed to an absolute path. In that case prefix the command with MSYS_NO_PATHCONV=1 like this (see Unable to clone repository after upgrading to Git v2.5.0).

MSYS_NO_PATHCONV=1 git tfs clone https://company.visualstudio.com:443 $/TFSProject/BranchName . --branches=none --export

Next I wanted to clean up the version history by removing the git-tfs metadata

git filter-branch -f --msg-filter "sed 's/^git-tfs-id:.*$//g'" -- --all

For some reason the NuGet packages had been stored in the version history. I decided to remove them to make the repository smaller.

#Remove packages folder from the history
git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch -r ./packages/*'
#Remove nuget folder from the history
git filter-branch -f --index-filter 'git rm --cached --ignore-unmatch -r ./.nuget/*'

I also had to remove TFS binding from the solution by removing the GlobalSection(TeamFoundationVersionControl) section from the solution (.sln) file. Finally I removed the branches that I didn’t need.

git branch -a
master
remotes/tfs/default
git branch -rd tfs/default

At this point I had a pretty clean Git repository. I created empty repository into VSTS and pushed the local repository into VSTS.

Configuration

On VSTS I configured repositories to use pull requests for develop and master branches and disabled push --force.

Use pull requests Disable push --force

First day after migration

First day the team run into following problems:

  • Not being able to see repositories (user connected to wrong TFS project)
  • Cannot push changes (user didn’t use pull request)
  • Fetch, Push, Pull confusion
  • VS UI confusion (creating branches, publishing changes)

Using Skype and screen sharing it was easy to help people to solve the minor problems they faced on the first day. It was great to see how team was able to transition from TFS to Git so easily!

Few weeks after migration

Few weeks after migration things are running smoothly. Sure there are times when someone makes changes to wrong branch or forgets to merge hotfix back to develop but this is expected. We are only humans. The team has already solved merge conflicts, released new versions, made hotfixes into production and used cherry pick!

Only change we made into the way we work is that DevOps doesn’t have to use pull requests. This is mainly because DevOps will create releases by merging code from release preparation branch into master branch. That code has already been reviewed.

In conclusion I would say the migration was successful and well worth the effort!