As our application evolved, we found ourselves frequently needing to update our MongoDB schema to accommodate new features. However, our first attempts at data migration resulted in some painful lessons. In one particularly stressful incident, we lost critical user preference data while attempting a seemingly simple field rename. This experience taught us to approach MongoDB migrations with much more caution and structure.

The Challenge with Schema-less Databases

One of MongoDB's great strengths is its flexible, schema-less nature. This flexibility, however, can become a double-edged sword when you need to evolve your data structure. Without enforced schemas, there's no built-in mechanism to ensure data consistency during structure changes.

Our initial approach to migrations was far too casual. We would write quick update scripts and run them directly against production:

// A typical early migration attempt
db.users.updateMany(
  {}, 
  { $rename: { "preferences": "userPreferences" } }
)

This approach led to several problems:

1. No way to test migrations beforehand
2. No transaction support for atomic updates
3. No rollback mechanism if something went wrong
4. Difficult to track which migrations had been applied
5. Hard to collaborate on migrations across the team

The Disaster: Silent Data Loss

The incident that changed our approach happened during what seemed like a simple migration. We needed to restructure our user preferences from a flat structure to a nested one:

// Original document structure
{
  "_id": ObjectId("5f8a7b9c3e24c2001f7a9b23"),
  "email": "[email protected]",
  "preferences": {
    "theme": "dark",
    "notifications": true,
    "language": "en"
  }
}

// Desired new structure
{
  "_id": ObjectId("5f8a7b9c3e24c2001f7a9b23"),
  "email": "[email protected]",
  "settings": {
    "ui": {
      "theme": "dark"
    },
    "communications": {
      "notifications": true,
      "language": "en"
    }
  }
}

We wrote what we thought was a clever migration script:

// Our flawed migration script
db.users.updateMany(
  {}, 
  {
    $set: {
      "settings.ui.theme": "$preferences.theme",
      "settings.communications.notifications": "$preferences.notifications",
      "settings.communications.language": "$preferences.language"
    },
    $unset: { "preferences": "" }
  }
)

When we ran this script, it appeared to work - no errors were reported. But soon users began reporting that their preferences had been reset to defaults. When we investigated, we discovered two critical mistakes:

1. We had used string literals for the values (with $ prefix) instead of properly referencing the existing fields with the $set operator
2. We had dropped the old field before verifying the new fields were populated correctly

The result was that all user preferences were replaced with literal strings like "$preferences.theme" rather than the actual values. Worse, we had no easy way to recover the lost data since we had already removed the original fields.

Lesson #1: Always Use Aggregation Pipeline for Complex Migrations

Our first major lesson was to use MongoDB's aggregation pipeline for update operations. This provides much more control and expressiveness:

// Better approach using aggregation pipeline
db.users.updateMany(
  {}, 
  [
    {
      $set: {
        "settings": {
          "ui": {
            "theme": "$preferences.theme"
          },
          "communications": {
            "notifications": "$preferences.notifications",
            "language": "$preferences.language"
          }
        }
      }
    }
  ]
)

Note the key differences:

• We used an array to define an aggregation pipeline
• The $ syntax now correctly references existing fields rather than creating string literals
• We didn't immediately drop the old fields

Lesson #2: Always Backup Before Migrating

After our data loss incident, we established a strict rule: always create a backup before running any migration. This could be as simple as:

// Create a backup collection before migration
db.users.aggregate([
  { $match: {} },
  { $out: "users_backup_" + new Date().toISOString().replace(/:/g, '_') }
])

For larger collections, we would use MongoDB's native tools:

# Using mongodump for larger collections
mongodump --uri="mongodb://user:password@host:port/db" --collection=users --out=./backup-$(date +%Y-%m-%d)

Lesson #3: Develop a Migration Framework

To address the structural problems with our migration approach, we developed a proper migration framework. We decided to build it with Node.js since that matched our application stack:

// migrations/framework.js
const { MongoClient } = require('mongodb');
const fs = require('fs');
const path = require('path');

class MigrationFramework {
  constructor(uri, options = {}) {
    this.uri = uri;
    this.options = options;
    this.dbName = options.dbName || 'myapp';
    this.migrationsCollection = options.migrationsCollection || 'migrations';
    this.migrationsDir = options.migrationsDir || path.join(__dirname, 'scripts');
    this.client = null;
    this.db = null;
  }

  async connect() {
    this.client = new MongoClient(this.uri);
    await this.client.connect();
    this.db = this.client.db(this.dbName);
    console.log(`Connected to database: ${this.dbName}`);
  }

  async close() {
    if (this.client) {
      await this.client.close();
      console.log('Database connection closed');
    }
  }

  async getAppliedMigrations() {
    const migrationsCollection = this.db.collection(this.migrationsCollection);
    return await migrationsCollection.find({})
      .sort({ appliedAt: 1 })
      .toArray();
  }

  async markMigrationAsApplied(migrationName, success, logs) {
    const migrationsCollection = this.db.collection(this.migrationsCollection);
    await migrationsCollection.insertOne({
      name: migrationName,
      appliedAt: new Date(),
      success,
      logs
    });
  }

  async getPendingMigrations() {
    const appliedMigrations = await this.getAppliedMigrations();
    const appliedMigrationNames = appliedMigrations.map(m => m.name);
    
    // Read migration scripts from directory
    const files = fs.readdirSync(this.migrationsDir)
      .filter(file => file.endsWith('.js'))
      .sort();
    
    return files
      .filter(file => !appliedMigrationNames.includes(path.parse(file).name));
  }

  async applyMigration(migrationName) {
    const filePath = path.join(this.migrationsDir, `${migrationName}.js`);
    const migration = require(filePath);
    
    console.log(`Applying migration: ${migrationName}`);
    
    const logs = [];
    const logCapture = (message) => {
      const logMessage = `[${new Date().toISOString()}] ${message}`;
      logs.push(logMessage);
      console.log(logMessage);
    };
    
    try {
      // Create backup if specified
      if (migration.backup) {
        const backupCollectionName = `${migration.backup}_backup_${Date.now()}`;
        logCapture(`Creating backup of ${migration.backup} to ${backupCollectionName}`);
        
        await this.db.collection(migration.backup).aggregate([
          { $match: {} },
          { $out: backupCollectionName }
        ]).toArray();
        
        logCapture(`Backup created: ${backupCollectionName}`);
      }
      
      // Apply the migration
      await migration.up(this.db, logCapture);
      
      // Mark as applied
      await this.markMigrationAsApplied(migrationName, true, logs);
      
      logCapture(`Migration ${migrationName} applied successfully`);
      return true;
    } catch (error) {
      logCapture(`Error applying migration ${migrationName}: ${error.message}`);
      
      // Log the failed attempt
      await this.markMigrationAsApplied(migrationName, false, [
        ...logs,
        `Error: ${error.message}`,
        `Stack: ${error.stack}`
      ]);
      
      throw error;
    }
  }

  async runMigrations() {
    try {
      await this.connect();
      
      const pendingMigrations = await this.getPendingMigrations();
      
      if (pendingMigrations.length === 0) {
        console.log('No pending migrations to apply.');
        return true;
      }
      
      console.log(`Found ${pendingMigrations.length} pending migrations`);
      
      for (const migrationFile of pendingMigrations) {
        const migrationName = path.parse(migrationFile).name;
        await this.applyMigration(migrationName);
      }
      
      console.log('All migrations applied successfully.');
      return true;
    } catch (error) {
      console.error('Migration failed:', error);
      return false;
    } finally {
      await this.close();
    }
  }
}

module.exports = MigrationFramework;

This framework provided critical features:

• Tracking of which migrations had been applied
• Automatic backup of affected collections
• Detailed logging of migration steps
• A consistent structure for migration scripts

Lesson #4: Structure Your Migration Scripts

With our framework in place, we created a standard structure for migration scripts:

// migrations/scripts/20230315_restructure_user_preferences.js
module.exports = {
  description: "Restructure user preferences into settings object",
  backup: "users", // Collection to backup before migration
  
  async up(db, log) {
    log("Starting user preferences restructuring");
    
    // Get count before migration for verification
    const countBefore = await db.collection('users').countDocuments({
      "preferences": { $exists: true }
    });
    log(`Found ${countBefore} users with preferences to migrate`);
    
    // Perform the migration using aggregation pipeline
    const result = await db.collection('users').updateMany(
      { "preferences": { $exists: true } },
      [
        {
          $set: {
            "settings": {
              "ui": {
                "theme": "$preferences.theme"
              },
              "communications": {
                "notifications": "$preferences.notifications",
                "language": "$preferences.language"
              }
            }
          }
        }
      ]
    );
    
    log(`Updated ${result.modifiedCount} documents`);
    
    // Verify all documents were migrated correctly
    const migratedUsers = await db.collection('users').countDocuments({
      "settings.ui.theme": { $exists: true },
      "settings.communications.notifications": { $exists: true },
      "settings.communications.language": { $exists: true }
    });
    
    if (migratedUsers !== countBefore) {
      throw new Error(`Migration verification failed: Expected ${countBefore} users to be migrated, but only ${migratedUsers} have the new structure`);
    }
    
    log("Verification passed: All preferences migrated correctly");
    
    // Only after verification, remove the old field
    const cleanupResult = await db.collection('users').updateMany(
      { "preferences": { $exists: true } },
      { $unset: { "preferences": "" } }
    );
    
    log(`Removed old preferences field from ${cleanupResult.modifiedCount} documents`);
  }
};

The key elements of this structured approach:

1. Clear description of what the migration does
2. Explicit specification of which collection to backup
3. Pre-migration count to verify against
4. Verification step before removing old data
5. Detailed logging throughout the process

Lesson #5: Test Migrations on Realistic Data

Another painful lesson we learned was the importance of testing migrations on realistic data. We developed a workflow for safely testing migrations:

// migration-test.js
const MigrationFramework = require('./framework');
const { MongoClient } = require('mongodb');
require('dotenv').config();

async function createTestEnvironment() {
  // Connect to production (read-only)
  const prodClient = new MongoClient(process.env.PROD_MONGODB_URI);
  await prodClient.connect();
  const prodDb = prodClient.db();
  
  // Connect to test environment
  const testClient = new MongoClient(process.env.TEST_MONGODB_URI);
  await testClient.connect();
  const testDb = testClient.db();
  
  console.log('Creating test environment with production data sample...');
  
  // Get list of collections to sample
  const collections = await prodDb.listCollections().toArray();
  
  // For each collection, copy a sample to test environment
  for (const collection of collections) {
    const collName = collection.name;
    if (collName.startsWith('system.')) continue;
    
    // Sample up to 1000 documents
    const documents = await prodDb.collection(collName)
      .find({})
      .limit(1000)
      .toArray();
    
    if (documents.length > 0) {
      // Drop existing test collection if it exists
      await testDb.collection(collName).drop().catch(() => {});
      
      // Insert the sampled documents
      await testDb.collection(collName).insertMany(documents);
      console.log(`Copied ${documents.length} documents from ${collName}`);
    }
  }
  
  await prodClient.close();
  await testClient.close();
  console.log('Test environment created successfully');
}

async function testMigration() {
  await createTestEnvironment();
  
  const migrationFramework = new MigrationFramework(
    process.env.TEST_MONGODB_URI,
    { 
      dbName: 'test',
      migrationsDir: './migrations/scripts'
    }
  );
  
  console.log('Running migrations on test data...');
  await migrationFramework.runMigrations();
  console.log('Test migration completed');
}

testMigration().catch(console.error);

This approach allowed us to safely test migrations on a subset of real production data, catching issues before they affected production.

Lesson #6: Use Transactions When Possible

For MongoDB 4.0+, we found that using transactions greatly improved the reliability of our migrations. Transactions ensure that multiple operations either all succeed or all fail together:

// Example migration with transaction support
module.exports = {
  description: "Update user account types",
  backup: "users",
  
  async up(db, log) {
    log("Starting user account type update");
    
    const session = db.client.startSession();
    
    try {
      session.startTransaction();
      
      // Create a new collection with the transformed data
      await db.collection('users').aggregate([
        { 
          $set: {
            accountType: {
              $switch: {
                branches: [
                  { case: { $eq: ["$plan", "free"] }, then: "BASIC" },
                  { case: { $eq: ["$plan", "premium"] }, then: "PREMIUM" },
                  { case: { $eq: ["$plan", "business"] }, then: "ENTERPRISE" }
                ],
                default: "BASIC"
              }
            }
          }
        },
        { $out: "users_new" }
      ], { session }).toArray();
      
      // Rename collections to swap the new one in place
      await db.collection('users').rename('users_old', { session });
      await db.collection('users_new').rename('users', { session });
      
      await session.commitTransaction();
      log("Transaction committed successfully");
    } catch (error) {
      await session.abortTransaction();
      log(`Transaction aborted: ${error.message}`);
      throw error;
    } finally {
      session.endSession();
    }
  }
};

When dealing with related data in multiple collections, transactions became even more important:

// Migration affecting multiple collections
module.exports = {
  description: "Update order references in both orders and users collections",
  backup: ["orders", "users"],
  
  async up(db, log) {
    // Only works with replica sets
    const session = db.client.startSession();
    
    try {
      session.startTransaction();
      
      // Update order format
      const orderResult = await db.collection('orders').updateMany(
        { format: "legacy" },
        { $set: { format: "new" } },
        { session }
      );
      
      log(`Updated ${orderResult.modifiedCount} orders`);
      
      // Update references in user collection
      const userResult = await db.collection('users').updateMany(
        { "orderReferences.format": "legacy" },
        { $set: { "orderReferences.$[elem].format": "new" } },
        { 
          arrayFilters: [{ "elem.format": "legacy" }],
          session 
        }
      );
      
      log(`Updated ${userResult.modifiedCount} user references`);
      
      await session.commitTransaction();
      log("Cross-collection update committed successfully");
    } catch (error) {
      await session.abortTransaction();
      log(`Transaction aborted: ${error.message}`);
      throw error;
    } finally {
      session.endSession();
    }
  }
};

Lesson #7: Handle Large Collections with Batching

As our application grew, we encountered collections with millions of documents. For these, we needed to implement batching to avoid overwhelming the database:

// Batched migration for large collections
module.exports = {
  description: "Add searchable field to all products",
  backup: "products",
  
  async up(db, log) {
    const batchSize = 1000;
    let processed = 0;
    let hasMore = true;
    let lastId = null;
    
    log("Starting batched migration of products");
    
    while (hasMore) {
      // Build query for current batch
      const query = lastId ? { _id: { $gt: lastId } } : {};
      
      // Get batch of documents
      const products = await db.collection('products')
        .find(query)
        .sort({ _id: 1 })
        .limit(batchSize)
        .toArray();
      
      if (products.length === 0) {
        hasMore = false;
        continue;
      }
      
      // Update logic for each document in the batch
      const bulkOps = products.map(product => ({
        updateOne: {
          filter: { _id: product._id },
          update: {
            $set: {
              searchableText: `${product.name} ${product.description} ${product.category || ''}`.toLowerCase()
            }
          }
        }
      }));
      
      // Execute batch update
      const result = await db.collection('products').bulkWrite(bulkOps);
      
      processed += products.length;
      lastId = products[products.length - 1]._id;
      
      log(`Processed batch of ${products.length} products. Total: ${processed}`);
    }
    
    log(`Completed migration of ${processed} products`);
  }
};

This batching approach had several advantages:

• Reduced memory consumption for large collections
• Limited the impact on database performance
• Allowed for better progress tracking
• Provided resilience if a migration was interrupted

Automating Migrations in CI/CD

Once we had built a reliable migration framework, we integrated it into our CI/CD pipeline. Migrations would run automatically after deployment, ensuring that the database schema always matched the application code:

# GitHub Actions workflow example
name: Deploy and Migrate

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      
      - name: Setup Node.js
        uses: actions/setup-node@v2
        with:
          node-version: '16'
      
      - name: Install dependencies
        run: npm ci
      
      - name: Build and deploy
        run: npm run deploy
      
      - name: Run database migrations
        run: node migrations/run.js
        env:
          MONGODB_URI: ${{ secrets.PROD_MONGODB_URI }}
          NODE_ENV: production

We also added a rollback capability, which required writing down migration operations:

// Migration with down function
module.exports = {
  description: "Add user.fullName field based on first/last name",
  backup: "users",
  
  async up(db, log) {
    log("Adding fullName field to users");
    
    const result = await db.collection('users').updateMany(
      { firstName: { $exists: true }, lastName: { $exists: true } },
      [
        {
          $set: {
            fullName: { $concat: ["$firstName", " ", "$lastName"] }
          }
        }
      ]
    );
    
    log(`Added fullName to ${result.modifiedCount} users`);
  },
  
  async down(db, log) {
    log("Removing fullName field from users");
    
    const result = await db.collection('users').updateMany(
      { fullName: { $exists: true } },
      { $unset: { fullName: "" } }
    );
    
    log(`Removed fullName from ${result.modifiedCount} users`);
  }
};

Conclusion: Our Final Migration Strategy

After several painful lessons and iterative improvements, our final migration strategy included:

1. Structured migration scripts with clear descriptions and rollback capabilities
2. Automatic backups of affected collections before each migration
3. Verification steps to confirm migrations completed successfully
4. Batching for large collections to manage performance impact
5. Transactions for operations that span multiple collections
6. Thorough testing on sample production data before deployment
7. Integration with CI/CD to automate migrations with deployments
8. Detailed logging for troubleshooting and auditing

This comprehensive approach has served us well, allowing us to confidently evolve our database schema alongside our application without the stress and risk of our early migration attempts. While it required more upfront investment, the peace of mind and prevention of data loss has more than justified the effort.

Remember, in the world of database migrations: measure twice, cut once. Your data is too valuable to risk with ad-hoc migration scripts.