Shell Script for Pushing Large Git Repositories

Check Out My Results

My requirement: I want to read Linux source code, manage my Linux source code reading with git, and save my source code reading with my self-built GitLab. Then I discovered that simply using git push for a 2GB Linux source code file would cause problems.

So I wrote a script to solve this problem.

The script content is as follows:

shellCopy
#!/bin/bash

# Configuration parameters (modify as needed)
REMOTE_NAME="origin"              # Remote repository name
BRANCH_NAME="main"                # Target branch name
REMOTE_URL="git@gitlab.cheverjohn.me:CheverJohn/linux.git"  # Remote repository URL
BATCH_SIZE=500                    # Number of files per batch
COMMIT_MESSAGE="Batch file upload"      # Commit message

# Color configuration
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No color

# Check if we're inside a git repository
if ! git rev-parse --is-inside-work-tree > /dev/null 2>&1; then
    echo -e "${YELLOW}Current directory is not a git repository, initializing...${NC}"
    git init
    echo -e "${GREEN}Git repository initialized${NC}"
else
    echo -e "${GREEN}Git repository already exists${NC}"
fi

# Check if remote repository is configured
if ! git remote | grep -q "$REMOTE_NAME"; then
    echo -e "${YELLOW}Adding remote repository $REMOTE_NAME: $REMOTE_URL${NC}"
    git remote add $REMOTE_NAME $REMOTE_URL
    echo -e "${GREEN}Remote repository added${NC}"
else
    CURRENT_URL=$(git remote get-url $REMOTE_NAME 2>/dev/null || echo "")
    if [ "$CURRENT_URL" != "$REMOTE_URL" ]; then
        echo -e "${YELLOW}Updating remote repository URL: $REMOTE_URL${NC}"
        git remote set-url $REMOTE_NAME $REMOTE_URL
        echo -e "${GREEN}Remote repository URL updated${NC}"
    else
        echo -e "${GREEN}Remote repository correctly configured${NC}"
    fi
fi

# Ensure main branch exists
if ! git show-ref --quiet refs/heads/$BRANCH_NAME; then
    echo -e "${YELLOW}Creating branch $BRANCH_NAME...${NC}"
    git checkout -b $BRANCH_NAME
    echo -e "${GREEN}Branch $BRANCH_NAME created${NC}"
else
    echo -e "${GREEN}Branch $BRANCH_NAME already exists${NC}"
    git checkout $BRANCH_NAME
fi

# Get all untracked and modified files
echo -e "${BLUE}Getting list of files to upload...${NC}"
FILES=($(git ls-files --others --exclude-standard) $(git diff --name-only))
TOTAL_FILES=${#FILES[@]}

if [ $TOTAL_FILES -eq 0 ]; then
    echo -e "${YELLOW}No files found to add, trying to add all files...${NC}"
    git add -A
    FILES=($(git diff --name-only --cached))
    TOTAL_FILES=${#FILES[@]}

    if [ $TOTAL_FILES -eq 0 ]; then
        echo -e "${RED}Error: No files found to push${NC}"
        exit 1
    fi
else
    # Add all files to staging area
    echo -e "${BLUE}Adding all files to staging area...${NC}"
    git add -A
fi

echo -e "${GREEN}Found $TOTAL_FILES files to upload${NC}"

# Calculate number of batches
BATCH_COUNT=$(( ($TOTAL_FILES + $BATCH_SIZE - 1) / $BATCH_SIZE ))
echo -e "${GREEN}Will upload in $BATCH_COUNT batches${NC}"

# First commit all files
echo -e "${BLUE}Committing all files...${NC}"
git commit -m "$COMMIT_MESSAGE"

# Push in batches
echo -e "${BLUE}Starting batch push...${NC}"
git push -u $REMOTE_NAME $BRANCH_NAME

if [ $? -eq 0 ]; then
    echo -e "${GREEN}All files successfully pushed to remote repository${NC}"
else
    echo -e "${YELLOW}Regular push failed, trying batch push method...${NC}"
    # Using git batch push method (referenced from search results examples)
    # Based on git_batch_push.sh concept, but simplified implementation

    # Use git rev-list to get all commits
    COMMITS=($(git rev-list --reverse HEAD))
    TOTAL_COMMITS=${#COMMITS[@]}

    if [ $TOTAL_COMMITS -eq 0 ]; then
        echo -e "${RED}Error: No commits found${NC}"
        exit 1
    fi

    echo -e "${GREEN}Found $TOTAL_COMMITS commits, will push in batches${NC}"

    # Calculate number of batches (500 commits per batch)
    COMMIT_BATCH_SIZE=500
    COMMIT_BATCH_COUNT=$(( ($TOTAL_COMMITS + $COMMIT_BATCH_SIZE - 1) / $COMMIT_BATCH_SIZE ))

    echo -e "${GREEN}Will push commits in $COMMIT_BATCH_COUNT batches${NC}"

    # Push commits in batches
    for ((i=0; i<$COMMIT_BATCH_COUNT; i++)); do
        START=$(($i * $COMMIT_BATCH_SIZE))
        END=$((($i + 1) * $COMMIT_BATCH_SIZE))

        if [ $END -gt $TOTAL_COMMITS ]; then
            END=$TOTAL_COMMITS
        fi

        BATCH_END_INDEX=$((END - 1))
        TARGET_COMMIT=${COMMITS[$BATCH_END_INDEX]}

        echo -e "${BLUE}Pushing batch $((i+1))/${COMMIT_BATCH_COUNT} (commits $((START+1))-$END)...${NC}"

        if git push $REMOTE_NAME $TARGET_COMMIT:refs/heads/$BRANCH_NAME; then
            echo -e "${GREEN}Batch $((i+1)) successfully pushed${NC}"
        else
            echo -e "${RED}Batch $((i+1)) push failed${NC}"
            echo -e "${YELLOW}Trying alternative push method...${NC}"

            # If above method fails, try another batch push method
            if [ $i -eq 0 ]; then
                # First batch, create new branch
                git push $REMOTE_NAME $BRANCH_NAME
            else
                # Get the end commit of previous batch
                PREV_END=$(($START - 1))
                PREV_COMMIT=${COMMITS[$PREV_END]}
                CURR_COMMIT=${COMMITS[$BATCH_END_INDEX]}

                echo -e "${BLUE}Using range push $PREV_COMMIT..$CURR_COMMIT${NC}"
                git push $REMOTE_NAME $PREV_COMMIT:refs/heads/$BRANCH_NAME $CURR_COMMIT:refs/heads/$BRANCH_NAME
            fi
        fi

        echo
    done
fi

echo -e "${GREEN}Batch upload process completed!${NC}"

The usage method is simple.

shellCopy
chmod +x git_batch_push.sh

Then run the script in the current folder.

Final Results