Extract project libraries to their own git repositories


I often find myself developing code for a project which over the course of the job turns out to be a useful general library which I'd like to use for other projects. Rather than copy/pasting this code to a new folder and starting a fresh Git repository, I'll want to take the commit history for that library and its subfolders, and start with that.

Based on this Stack Overflow post, there are a couple of approaches to do that:

  1. Extract only the folder and subfolders
  2. Extract the full path (if you have a root folder structure you want to keep)

For now I'm going to demonstrate the first.

There's a lot of complicated sounding information out there, which often assumes you know what you're doing on the git command line (which I don't) so:

  • I'll use an actual project to illustrate how this is done
  • I'll break it down in super-easy steps, with full code output, so you can see what's happening
  • I'll highlight the code you type in red
  • I'll underline the names of folders and repositories to make it clear what you should swap out for your own values, and what's just a git command

Note that this was done on a mac, so the paths are normal, absolute paths.

  • Windows users, use the standard path syntax for Windows, c:/path/to/files
  • All users, use quotes for pathnames with spaces: "c:/path/to/some folder/with lots of files"

Technique 1: Extract only folder and subfolders


This technique results in a brand new repo, with the subfolder contents in the root:

+- original                 +- some-lib
    +- blah                     +- ClassA
    +- libs                     +- ClassB
        +- lib1                 +- ClassC
        +- lib2
        +- lib3         ->
            +- ClassA
            +- ClassB
            +- ClassC

This is useful in languages like PHP, when extracting a library to use with Composer, as Composer recreates the intermediate namespacing folders (e.g. davestewart/some-lib/) when you require the package from Packagist.

We use git subtree split to copy the folder into a new branch in the original repo, which you then pull into a new repo.


To start, we'll create a temporary branch in your original project repo.

Go to the original repo's folder (path/to/original-repo) and split the library subfolder (path/to/lib-folder) into a new branch (some-lib):

$ cd /path/to/original-repo
$ git subtree split -P path/to/lib-folder -b some-lib
Created branch 'some-lib'

Next, we'll make a new repo and pull in the temporary branch.

Create and initialize a new repo (/path/to/new-repo):

$ mkdir /path/to/new-repo
$ cd /path/to/new-repo
$ git init
Initialized empty Git repository in /path/to/new-repo/.git/

The final step is to grab the branch you created a moment ago.

From the new repo (current folder) pull the newly-created branch (some-lib) from the original repo (/path/to/original-repo):

$ git pull /path/to/original-repo some-lib
remote: Counting objects: 235, done.
remote: Compressing objects: 100% (217/217), done.
remote: Total 235 (delta 90), reused 16 (delta 3)
Receiving objects: 100% (235/235), 61.53 KiB | 0 bytes/s, done.
Resolving deltas: 100% (90/90), done.
From /path/to/original-repo
 * branch            crud-lib   -> FETCH_HEAD

Do a quick file listing, just to see if it's worked:

$ ls -l
total 536
drwxr-xr-x@ 1 Dave  staff   189B 30 Mar 17:14 docs/
drwxr-xr-x@ 1 Dave  staff   756B 30 Mar 17:14 src/
drwxr-xr-x@ 1 Dave  staff   504B 30 Mar 17:14 views/

You can check the logs as well, just to see that it is only the commit history from this folder:

$ git shortlog
Dave Stewart (37):
      Moved crud helper files to their own namespace
      Moved local packages around
      Major rewrite of CrudHelper to CrudService
      Updated CrudService with explicit getters and additional flexibility to set values and use the class flexibly
      Various improvements and optimisations on CrudService package
      Minor updates to crud view data structures
      Various minor Crud updates
      Various improvements to CRUD views and structure, and moved Entity Controllers to controllers/entities folder
      Added proper index redirecting
      Moved all querying to internal query builder, broke out call() and query() methods, added filter() method, added magic properties and method calls
      Updated CrudService to work with updated CrudRepo
      Updated CrudMeta behaviour for index pages
      Separated CrudRepo functionality into standalone methods, as well as various other tidy ups and comments
      Added robust CrudMeta getter and setter methods
      Made CrudService::index() method internals more readable
      Moved CrudMethods trait to traits subfolder
      Converted CrudRepo interface to base class, and made some minor code optimisations in EloquentRepo
      Added control() helper to CrudServiceProvider
      Various major optimisations to CrudService, CrudMata and CrudField
      Added CrudControl and FormControl classes
      Added new exception class
      Added field-centric validation messages
      Moved all field-generation functionality to CrudMetaService
      Moved and minor updates to CrudService
      Minor bug fixes to CrudMeta/CrudMetaService refactor
      Added back rules, and added required class to control classes
      Moved stub view files
      Renamed CRUD validation messages file
      Added confirmation messages to crud messages
      First iteration of CrudValidator
      Updated language files
      Added basics of translation service
      Reorganised CrudMeta and removed status / language functionality
      Found cleaner way for CrudValidator to display proper messages
      Various major improvements in fields and controls to add more flexibility and functionality
      Moved loadRelated() functionality from CrudService to CrudRepo
      Various improvements to CrudMetaService

That's all there is to it 🙂

If you're writing PHP, and still developing your package, see this post to see how to use Composer to manage and include your new package whilst at the same time continuing local development on it.

Technique 2: Extract folder and subfolders


This technique results in a brand new repo, with the the entire, filtered subtree in the root:

+- original                 +- some-lib
    +- blah                     +- libs
    +- libs                         +- lib3
        +- lib1                         +- ClassA
        +- lib2                         +- ClassB
        +- lib3         ->              +- ClassC
            +- ClassA
            +- ClassB
            +- ClassC

This is useful for languages like ActionScript, where you need the full folder structure in order to find the classes by folder.


I'll cover this at some point, but in the meantime, look at the second option covered in the Stack Overflow link below.


I used the following resources to compile this post:

Leave a Comment