From the exquisite penmanship of Mihai Badila
The problem
Recently a challenge we encountered while developing a website builder involved user-generated content that was saved in our database and then used to render user pages. This process was done by offering users the possibility to create a website by dragging and dropping some predefined components.
The core issue at hand was the process required whenever component properties were altered. Each modification demanded a manual reset, a task both time-consuming and prone to human error. This not only hindered the efficiency of our development process but also impacted the overall user experience of the website builder since users had to manually go back into their page’s admin panel and either reset that component or modify its settings based on the new updates.
This article delves into our journey to devise an automated solution for migrating these components, ensuring seamless updates and consistency.
Path to the Solution
Before we agreed to develop this migration CLI we first went through a round of research to see which other solutions are available or what else can we do to overcome this.
Initially, our approach was to create a series of fallback mechanisms for every update introduced. This required us to establish default values and establish connections between the legacy and updated properties whenever we altered a component’s characteristics. However, this strategy had significant downsides. Over time, it led to an accumulation of redundant properties and resulted in a codebase that could become increasingly complex and challenging to manage.
Consequently, following further exploration, we settled on a strategy where we could author a script for which we can define migrations to accompany each component update. This script would encompass a new migration each, set to be executed across all pages in the database. In the event of any discrepancies, we have the option to revert to the earlier version by calling the ‘migrate down’ method. This is why having the migrate-down and migrate-test defined is important, we can test before deploying and rollback in case of emergency.
Implementation
After finalizing our strategy, we focused on the implementation phase. The goal of our solution was the development of a migration Command Line Interface (CLI). This tool was designed to handle the up and down migrations of any JSON schema and test its consistency.
Prerequesites
For this, we had to think of the following methods:
- generateMigration – this method is pretty straight forward, it will generate a template file with the version given by a timestamp and contains declaration and export for 2 methods, up respectively down.
- runMigrations – Based on given arguments it will read all the migrations from a specified folder and run the given JSON data until it meets the version we want it to be.
- runMigrationTest – a method for testing. It will allow us to take a JSON object run all up migrations then roll it back down to the version we started with and check if the data remains consistent.
Creating the CLI
Create an empty project using the package manager of your choice. For this example we will stick to npm.
- For this prohect we will use typescript thus you can run
- Install the
commanderin your project directory. - We can create a new
scripts.tsfile that will hold our methods and acli.tsthat will take care of the commands that will call methods from the scripts file.
> npm i -D @types/node typescript ts-node
> npm i commander
Generating migrations
We will start with generateMigration. This method has the job of creating a new file inside the given directory with a template content for up and down methods.
// scripts.ts
export const generateMigration = async (
customPath: string | undefined,
name: string | undefined
) => {
const timestamp = new Date().getTime();
const migrationName = `${timestamp}-${name || "migration"}.ts`;
const migrationTemplate = `
// ${migrationName}
export async function up(data: any) {
// Add your migration logic here
}
export async function down(data: any) {
// Add your rollback logic here
}
`;
const migrationDir = path.join(__dirname, customPath || BASE_MIGRATIONS_PATH);
const migrationFilePath = path.join(migrationDir, migrationName);
if (!fs.existsSync(migrationDir)) {
fs.mkdir(migrationDir, (err) => {
if (err) throw err;
});
}
await fs.writeFile(migrationFilePath, migrationTemplate, (err) => {
if (err) throw err;
});
console.log(`New migration file created: ${migrationFilePath}`);
};
Running Migrations
Now after the users generate a new migration and modify the object according to their schema update, we will need a way to run our data iteratively through all of those. This is how runMigrations comes into play.
customPath is a string that specifies the path to the directory containing the migration files. If it is not provided, the default path is used.
direction is a string that specifies the direction of the migration. It can be either “up” or “down”.
from and to parameters are strings that specify the starting and ending versions of the migration. They can be either a version number or the strings “0” or “latest”, that represent the first and respectively latest version.
jsonData which is an object that represents the data that will be migrated.
// scripts.ts
export const runMigrations = async (
customPath: string,
direction: "up" | "down",
from: string,
to: string,
jsonData: any
) => {
// Read all files from the specified folder and sort them by name.
const migrationFiles = fs
.readdirSync(path.join(__dirname, customPath))
.filter((file) => file.endsWith(".ts"))
.sort();
// Since we sorted them previously by timestamp
// we know that for running down migrations
// we can only reverse the order and iterate through them
if (direction === "down") migrationFiles.reverse();
for (const migrationFile of migrationFiles) {
// Import up and down methods from current migration
const { up, down } = require(path.join(
__dirname,
"migrations",
migrationFile
));
// Get the version so we know the version we are on
const version = migrationFile.split("-")[0];
const shouldMigrationRun = false
// Verify if we need to run the current migration.
if (direction === "up") {
if (from === "0" && to === "latest") shouldMigrationRun = true;
if (from === "0" && version <= to) shouldMigrationRun = true;
if (from <= version && to === "latest") shouldMigrationRun = true;
if (from <= version && version <= to) shouldMigrationRun = true;
}
if (direction === "down") {
if (from === "latest" && to === "0") shouldMigrationRun = true;
if (from === "latest" && version >= to) shouldMigrationRun = true;
if (from >= version && to === "0") shouldMigrationRun = true;
if (from >= version && version >= to) shouldMigrationRun = true;
}
if (shouldMigrationRun) {
// Run migration according to the given direction
if (direction === "up") {
console.log(`Migrating up: ${migrationFile}`);
await up(jsonData);
} else {
console.log(`Rolling back: ${migrationFile}`);
await down(jsonData);
}
}
}
// Return the updated JSON data
return jsonData;
};
Testing migrations
Now to be able to test each migration we will define the runMigrationTest method. To check to JSON objects without taking into consideration the order of the properties we can use isEqual from lodash
// scripts.ts
export const runMigrationTest = async (
path: string,
from: string,
to: string,
originalJsonData: any
) => {
// Create a deep copy so we do not modify our original data.
const jsonData = { ...originalJsonData };
// Run all migration up to the given version and back to where we started.
await runMigrations(path, "up", from || "0", to || "latest", jsonData);
await runMigrations(path, "down", from || "latest", to || "0", jsonData);
// Verify consistency with the original JSON data
if (isEqual(jsonData, originalJsonData)) {
console.log("✅ Migration test passed. Data remains consistent.");
return true;
} else {
console.error("❌ Migration test failed. Data inconsistency detected.");
return false;
}
};
The CLI
To put everything together we will define a command for each of the above methods in the cli.ts file using comander like this:
// cli.ts
const program = new Command();
program
.command("migrate-up [path] [from] [to]")
.description("Migrate up from one version to another")
.action(async (migrationsPath: string, from: string = "0", to: string = "latest") => {
const jsonData = loadJsonData();
const newJsonData = await runMigrations(migrationsPath, "up", from, to, jsonData);
saveJsonData(newJsonData);
});
program
.command("migrate-down [path] [from] [to]")
.description("Migrate down from one version to another")
.action(async (migrationsPath: string, from: string = "latest", to: string = "0") => {
const jsonData = loadJsonData();
const newJsonData = await runMigrations(migrationsPath, "down", from, to, jsonData);
saveJsonData(newJsonData);
});
program
.command("migrate-test [path] [from] [to]")
.description("Migrate up and down for testing consistency")
.action(async (migrationsPath: string, from: string, to: string) => {
const jsonData = loadJsonData();
await runMigrationTest(migrationsPath, from, to, jsonData);
});
program
.command("migrate-new [path] [name]")
.description("Generate a new migration file")
.action(
async (migrationsPath: string | undefined, nameArg: string | undefined) => {
await generateMigration(migrationsPath, nameArg);
}
);
program.parse(process.argv);
The loadJsonData and SaveJsonData methods are utility methods to help us test the CLI, all they do is load a sample JSON file, respectively save a JS object to and .json file.
Hope you enjoyed this article, be sure to check out our other guides and articles on our blog and if you’re on the lookout for a team to help you with your digital products make sure to get in touch.
Recent Comments