Wtf!

My tests failed because Jest encountered an unexpected token; they failed because Jest cannot handle ESM-only packages out of the box. In fact, Jest is written in CommonJS. But what does that mean? To do so, we need to understand why CommonJS and ESM exist.

Why Do We Need Module Systems?

In the early days of web development, JavaScript was mainly used to manipulate the Document Object Model (DOM) with libraries like jQuery. However, the introduction of Node.js also led to JavaScript being used for server-side programming. This shift increased the complexity and size of JavaScript codebases. As a result, there arose a need for a structured method to organize and manage JavaScript code. Module systems were introduced to meet this need, enabling developers to divide their code into manageable, reusable units^1.

The Emergence of CommonJS

CommonJS was established in 2009, originally named ServerJS^2. It was designed for server-side JavaScript, providing conventions for defining modules. Node.js adopted CommonJS as its default module system, making it prevalent among backend JavaScript developers. CommonJS uses require to import and module.exports to export modules. All operations in CommonJS are synchronous, meaning each module is loaded individually.

The Rise of ESM (ECMAScript Modules)

In 2015, ECMAScript introduced a new module system called ECMAScript Modules (ESM), primarily targeting client-side development. ESM uses import and export statements, and its operations are asynchronous, allowing modules to be loaded in parallel^3. Initially, ESM was intended for browsers, whereas CommonJS was designed for servers. It became more and more a standard for the JS ecosystem. Nowadays, modern JavaScript runtimes support both module systems. Browsers began supporting ESM natively in 2017. Even Typescript adapted the ESM syntax, and whenever you learn it, you also learn ESM subconsciously.

CommonJS is here to stay

The truth is that there are many more CommonJS (CJS)- only packages than ESM-only packages^4.  However, there is a clear trend. The number of ESM-only or dual module packages is on the rise, while fewer CJS-only packages are being created. This trend underscores the growing preference for ESM and raises the question of how many of the CJS-only packages are actively maintained.

Comparison

An interesting comparison between CommonJS and ESM involves performance benchmarks. Due to its synchronous nature, CommonJS is faster when directly using require and import statements. Let's consider the following example:

// CommonJS -> s3-get-files.cjs
const s3 = require('@aws-sdk/client-s3');
new s3.S3Client({ region: 'eu-central-1' });

// ESM -> s3-get-files.mjs
import { S3Client } from '@aws-sdk/client-s3';

new S3Client({ region: 'eu-central-1' });

I used the aws-sdk S3-Client because it has dual module support. Here we instantiate the client and then execute it with node:

hyperfine --warmup 10 --style color 'node s3-get-files.cjs' 'node s3-get-files.mjs'

Benchmark 1: node s3-get-files.cjs
Time (mean ± σ): 82.6 ms ± 3.7 ms [User: 78.5 ms, System: 16.7 ms]
Range (min … max): 78.0 ms … 93.6 ms 37 runs
Benchmark 2: node s3-get-files.mjs
Time (mean ± σ): 93.9 ms ± 4.0 ms [User: 98.3 ms, System: 18.1 ms]
Range (min … max): 88.1 ms … 104.8 ms 32 runs

Summary
node s3-get-files.cjs ran
  1.14 ± 0.07 times faster than node s3-get-files.mjs

As you can see, the s3-get-files.cjs and, thus, CommonJS run faster. I got inspired by Buns Blogpost.

However, when you want to productionize your JS library, you need to bundle it. Otherwise, you will ship all the node_modules. Is used esbuild because it is able to bundle to CJS and ESM. Now, let's run the same benchmark with the bundled version.

hyperfine --warmup 10 --style color 'node s3-bundle.cjs' 'node s3-bundle.mjs'

Benchmark 1: node s3-bundle.cjs
Time (mean ± σ): 62.1 ms ± 2.5 ms [User: 53.8 ms, System: 6.7 ms]
Range (min … max): 59.5 ms … 74.5 ms 45 runs

Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Benchmark 2: node s3-bundle.mjs
Time (mean ± σ): 45.3 ms ± 2.2 ms [User: 38.1 ms, System: 5.6 ms]
Range (min … max): 43.0 ms … 59.2 ms 62 runs

Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet system without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

Summary

  node s3-bundle.mjs ran
    1.37 ± 0.09 times faster than node s3-bundle.cjs

As you can see, the s3-bundle.mjs is now faster than the s3-bundle.cjs. The ESM file is now even faster than the unbundled CommonJS file because it results in smaller file sizes and faster load times due to efficient tree-shaking—a process that removes unused code.

Embrace ESM!

The future of JavaScript modules is undoubtedly leaning towards ESM. This starts when creating a new NodeJS project or even a React project. Every tutorial and article uses the import—statement, which is thus ESM. Despite many existing CommonJS packages, the trend is shifting as more developers and maintainers adopt ESM for its performance benefits and modern syntax. Another question is also how many of these CJS-only projects are still maintained.

ESM is a standard that works in any runtime, such as NodeJS, Bun, or Deno, and in the browser without running on a server. It is unnecessary to convert via Babel to CommonJS because the browser understands ESM. You can still use Babel to convert to a different ECMAScript version, but you shouldn't convert to CJS.

You should develop in ESM-only because every runtime now and browser newer than 2017 understands ESM.

If your code breaks, you may have legacy issues. Consider using different tooling or packages. For example, you can migrate from Jest to vitest or from ExpressJS to h3. The syntax remains the same; the only difference is the import statement.

Key Takeaways:

• Smaller Bundles: ESM produces smaller bundles through tree-shaking, leading to faster load times.

• Universal Support: ESM is supported natively by browsers and JavaScript runtimes (Node.js, Bun, Deno).

• Future-Proof: With ongoing adoption, ESM is positioned as the standard for modern JavaScript modules.

To get started, you can follow this Gist or get an inspirational learning here.

For a better JavaScript Future, Embrace ESM!

The presentation

More Resources

• https://dev.to/logto/migrate-a-60k-loc-typescript-nodejs-repo-to-esm-and-testing-become-4x-faster-22-4a4k

• https://jakearchibald.com/2017/es-modules-in-browsers/

• https://gist.github.com/joepie91/bca2fda868c1e8b2c2caf76af7dfcad3

• https://gist.github.com/joepie91/bca2fda868c1e8b2c2caf76af7dfcad3

GHosttp is a tiny development server for developing GCP functions. With GHosttp, you can choose the folder where your GCP functions reside, allowing you to start developing from there. Your functions can be developed in Typescript and use the latest ESM features.

Why?

The functions framework by Google does not offer a great developer experience when developing an entire backend using GCP. Whenever you develop or test a new function, you must run the npx @google-cloud/functions-framework --target=myFunction command. Not only is that super long, but it also does not offer hot module reloading. What if the name changes? You need to rerun the command. Do you really want to create a bash alias for this library? Furthermore, their repository shows CommonJS syntax. What if you want to use ESM or Typescript? Since Gen2 is running on Node v20, you could and should leverage ESM. GHosttp is familiar with both ESM and Typescript.

Cloud Run

The Gen2 GCP cloud functions use ExpressJS on top of Cloud Run. ExpressJS comes with a development server. So, why not directly develop it locally in Express and then put it in a Dockerfile? Well, that's a lot of work, and you must be familiar with Docker and Cloud Run. GCP Cloud Functions gen2 abstracts many things for you.

Opinionated

This package is a bit opinionated about developing your GCP function, but I made it as close to the ExpressJS function as possible. You must name your function handler, and it shall be exported. Below you find a Typescript version.

import type { Request, Response } from '@google-cloud/functions-framework';

export const handler = async (req: Request, res: Response) => {
  res.set('Access-Control-Allow-Origin', '*');
  res.set('Access-Control-Allow-Methods', 'POST');

  // Because of CORS
  if (req.method === 'OPTIONS') {
    res.set('Access-Control-Allow-Methods', 'GET');
    res.set('Access-Control-Allow-Headers', 'Content-Type');
    res.set('Access-Control-Max-Age', '3600');
    res.status(204).send('');
    return;
  }

  // Destructering your request body
  const { message } = req.body;
  res.set('Content-Type', 'application/json');
  res.send(
    JSON.stringify({
      message,
    }),
  );
  return new Response(JSON.stringify(message));
};

Requirements

• GCP account

• GCP functions gen2

• Node >= 20 (I haven't tried with 18, but it's a good point to upgrade)

Getting Started

In your repository, just run npx ghosttp --dir path/to/my/functions. That's all. Per default, your backend will run on http://localhost:3000

  ➜ Local:    http://localhost:3000/
  ➜ Network:  use --host to expose

🚀 Loading server entry ./src/run-dev-server.ts GHosttp 12:21:58 AM
✅ Server initialized in 90ms                   GHosttp 12:21:58 AM
👀 Watching ./path/to/functions for changes     GHosttp 12:21:58 AM
ℹ Following endpoints are available             GHosttp 12:21:58 AM
ℹ /logger                                       GHosttp 12:21:58 AM
ℹ /run-dev-server                               GHosttp 12:21:58 AM

It watches whenever you make changes to your function or add a new function. This is great if you want to do Test-Driven-Development and get instant feedback.

Watch the video above to see it in action.

Arguments

• --dir - Path to your functions (default ".")

• --port - Port where your localhost will run (default 3000)

PR and contributions are welcome

Find the repository here: https://github.com/jolo-dev/ghosttp

What's next?

I will add a development server for Flask and Go. Furthermore, I want to add a new command, npx ghost add-handler my-new-handler --typescript, which scaffolds a new file. I will also make a blog/video about deploying your ESM/Typescript GCP function ;)

Let me know what you think, and I will be happy to make any contribution 🤩

Have you ever reviewed a PR that has formatting changes?

That could happen because developer tend to have their configurations and settings or don't use the linting and formatting command and simply skip the pre-commit 🫠

Let's say your team uses VS Code as their editor. What if you could use the .vscode folder to share repository settings? You can put settings, configuration, and recommended extensions inside .vscode and share them with your colleagues. This ensures that everyone is on the same page, reducing the chances of errors and inconsistencies. However, I have noticed that some people delete the .vscode folder or put it into the .gitignore.

But with that, you are missing out on its capabilities to streamline settings and even increase productivity by enabling formatting and linting. So, please Don't (.git)ignore the vscode 🙅‍♂️ Let's explore together how to utilize its features to take your development to the next level.

Understanding the Importance of Configuration

To create a consistent configuration, you can create a settings.json file in your VS Code folder. This file allows you to define settings for your project, such as the formatting style, compiler version, and default formatter. By sharing this file with your team, you can ensure everyone is working with the same configuration.

{
    'editor.formatOnSave': true,
    'typescript.tsdk': './node_modules/typescript/lib',
    '[typescript]': {
        'editor.defaultFormatter': 'biomejs.biome'
    },
    'eslint.format.enable': false,
    'typescript.format.enable': false,
    'prettier.enable': false
}

• editor.formatOnSave: true - allows formatting whenever you hit the save button

• typescript.tsdk - ensures using the correct compiler version

• editor.defaultFormatter - allows changing the formatter. You can have individual formatter by programming languages. Above we define a formatter for Typescript- files.

• eslint.format.enable: false - ensures that ESLint is disabled because, in my case, BiomeJS is able to do it. Otherwise, you may have conflicting linting errors

• typescript.format.enable: false - disable the VS Code in-built formatter. It could conflict with the default formatter

• prettier.enable: false - same as above

Recommending Extensions

VS Code allows you to recommend extensions to new users or team members. By adding an extensions.json file to your VS Code folder, you can specify the extensions that are required for your project. This ensures that new users have the necessary tools to work on the project efficiently.

{
  'recommendations': [
    'biomejs.biome'
  ]
}

Code Snippets

Code snippets are a powerful feature in VS Code that allows you to scaffold code quickly. By creating custom snippets, you can generate boilerplate code for your project, reducing the time and effort required to write code. However, the syntax for defining is quite complicated. That is why I would recommend to use the snippet generator tool which helps you create snippets quickly and efficiently.

{
  'Creates a simple Snippet': {
    'prefix': 'snippet',
    'description': 'Creates a simple Snippet',
    'body': [
      'import { http, type Request, type Response } from '@google-cloud/functions-framework';',
      '',
      'export const $0 = async ( req: Request, res: Response ) => {',
      '  const result = { result: 'This is gonna be the result for Handler: $0'}'
      '  res.send(',
      '    JSON.stringify( {',
      '      result,',
      '    } ),',
      '  );',
      '};'
    ]
  }
}

• prefix - the shortcut that appears in the autocompletion box

• description - the description of the snippet

• body- the code which will be generated or scaffolded

• $0- where the cursor will land. You can add multiple variables by $1, $2 etc. you change the variables by tabbing.

Launching a Fully Fledged IDE

VS Code can be turned into a fully-fledged IDE by using the launch.json file. This file allows you to define configurations for your project, including the executable, working directory, and runtime arguments. By using the launch.json file, you can create a customized development environment that suits your needs.

For example, when you have a Monorepo you can turn on a debugger for the backend and frontend.

{
    'configurations': [
        {
            'type': 'node',
            'name': 'Run Backend',
            'cwd': '${workspaceFolder}/packages/backend',
            'request': 'launch',
            'runtimeExecutable': 'bun',
            'runtimeArgs': [
                'dev'
            ],
            'console': 'integratedTerminal'
        },
        {
          'type': 'expo',
          'request': 'attach',
          'name': 'Debug Expo app',
          'projectRoot': '${workspaceFolder}/packages/app',
          'bundlerPort': '8081',
          'bundlerHost': '127.0.0.1'
        }
    ]
}

• type - Where the debugger should be running

• request - either attach for attaching it to a running instance e.g. you turn on the development server and want to attach the debugger to it, or launch where you create a new instance

• name - the name will be reflected in the Run and Debug tab

• cwd or projectRoot- change working directory if your script is located in a different folder

• runtimeExecutable - for running a binary

• runtimeArgs - the arguments that the binary is using

• console - where you want to log the logs. VS Code is giving you suggestions

• if attach, bundlerPort - the port where the current instance is running

• if attach, bundlerHost - the url where the current instance is running (usually, localhost)

Conclusion

In conclusion, VS Code is a powerful tool that can help you streamline your development process, improve code quality, and increase productivity. You can take your development to the next level by configuring your editor like a pro. Don't ignore the power of VS Code; instead, harness its features to create a customized development environment that suits your needs ⭐

Komiser

Komiser is an open-source platform loved by cloud experts and developers. Connect your cloud accounts to get detailed insights into your cloud infrastructure.

It provides a nice dashboard to see your cloud assets inventory, understand their resource dependencies, and uncover idle, underutilized, and untagged resources. Komiser aims to bring transparency to cloud management. I see it as a useful tool for creating an inventory of your cloud resources. It can be connected to AWS, GCP, Azure, Digital Ocean, and more. It is interesting to see what resources are abstracted from when using a library like AWS CDK or AWS SAM.

Steampipe

Steampipe is a tool for querying cloud APIs and other data sources using SQL in a zero-ETL manner.

Some key capabilities include:

• Dynamically query APIs, code repositories, logs, and other data sources using SQL for unified access.

• Combining data from different sources creates new insights through relationships and attributes across entities.

• It operates without needing to set up traditional databases through its "Zero-ETL" approach of querying data directly at the source.

• Providing metadata about available data tables and fields to help with the discovery and understanding of cloud resources and attributes.

I think Steampipe's dashboards (Mods) make Steampipe more powerful. They offer a variety of them, such as AWS Well-Architected, AWS Compliance, and many more.

If you look for an inventory, you can check how to build your dashboard.

StackQL

Like Steampipe's revolutionary approach, StackQL harnesses the power of SQL to query your resources seamlessly. Moreover, it empowers you to utilize SQL syntax for querying and creating resources.

INSERT INTO aws.s3.buckets (bucket, region)  
SELECT 'stackql-demo',  
'eu-central-1'

It is versatile and usable. Even though it is written in Go, they provide a Python package, which means you can use it with Jupyter Notebooks. I also like their guide on using it with Superset. In comparison to Steampipe, you can query all AWS services, whereas Steampipe is missing some. However, Steampipe offers autocompletion, which StackQL not have.

I used this instruction to create the inventory of my AWS account.

ChatWithCloud

Thanks to a generative AI model, ChatWithCloud allows users to interact with and manage their AWS cloud infrastructure using natural language commands in their terminal. Some key capabilities mentioned include:

• Performing cost analysis to find out spending and optimize costs

• Conducting security analysis by reviewing IAM policies and resources

• Troubleshooting infrastructure issues to understand problems and how to fix them

• Having the ability to fix infrastructure issues directly through commands

• There is no need for AWS knowledge or API keys to use most basic functions

So, in summary, ChatWithCloud provides a natural language interface for viewing, analyzing, and troubleshooting AWS resources with the help of an AI assistant. You must buy a license and use your OpenAI key or subscribe to it.

Conclusion

All these tools provide a comprehensive overview of your AWS resources. Komiser also displays costs, while ChatWithCloud provides answers by entering prompts. Steampipe and StackQL use SQL to interact with the cloud. Additionally, you can connect these to dashboards to visualize them, similar to any data analytics process.

Bonus: Create Inventory with AWS Console

Instructions are from here.

The tools above offer dynamic monitoring of your AWS resources. But what if you need inventory just once or less frequently? The following steps allow you to create a CSV of all the resources within your account:

• Use the Resource Group & Tag Editor.

  

• Choose the regions from which you want to fetch. The more regions you select, the longer it will take.

  

• Then download them.

  

CodePipeline, CodeBuild, CodeCommit! These are the DevOps services of AWS.

Your first reaction is, why? This looks so ugly! Where should I start? But the code is already on CodeCommit. It’s a monorepo.

Your research finds an interesting article on creating sophisticated pipelines entirely on AWS with AWS CDK. That blog post demonstrates each pipeline for different scenarios and how to organize your codebase. Let's have a look together.

Simple Pipeline

The simple pipeline includes a CodePipeline for orchestrating your CI/CD workflow.

A CodePipeline consists of stages, and each stage contains actions. An action is a command which is executed. Each stage produces an artifact resulting from an action that can be passed to the next stage. A stage always needs an artifact as an input, and it can define multiple actions that can run in parallel or sequentially.

Below is the AWS CDK code for the simple pipeline. Out of brevity, it left out the import statements.

// Create the pipeline
const pipeline = new Pipeline(stack, "Pipeline", {
  pipelineName: "aws-cdk-pipeline-demo",
});
// Create the artifact to store outputs of the pipeline
const artifact = new Artifact();
// 1.1. Best Practice to Import Repository
const repository = Repository.fromRepositoryName(
  stack,
  "Repository",
  "aws-cdk-pipeline-demo",
);
// 1.2. Add the source action to the pipeline
pipeline.addStage({
  stageName: "Source",
  actions: [
    new CodeCommitSourceAction({
      actionName: "Source",
      output: artifact,
      repository,
      branch: "main",
    }),
  ],
});
// 2. Stage : CodeBuild
pipeline.addStage({
  stageName: "Build",
  // We can actually paste many actions in here
  // But since we can dump anything in one CodeBuild
  actions: [
    new CodeBuildAction({
      actionName: "Build",
      input: artifact,
      project: new PipelineProject(stack, "BuildProject", {
        projectName: "aws-cdk-build-project",
        // You can also create your own buildspec.yml file and reference it here instead of using the inline buildspec
        // https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
        buildSpec: BuildSpec.fromObject({
          version: "0.2",
          phases: {
            install: {
              commands: ["npm install"],
            },
            build: {
              commands: [
                "npm run lint",
                "npm run test:unit",
                "npm run deploy",
                "npm run test:integration",
              ],
            },
          },
          artifacts: {
            "base-directory": ".",
            files: ["**/*"],
          },
        }),
      }),
    }),
  ],
});

However, the Simple Pipeline would not meet your requirements. Because you also want to deploy to different AWS accounts.

Cross-Account Pipeline

You continue reading, and indeed, there is a section about deploying to different AWS accounts. You learned that CodePipeline is made with that in mind. Below, you see the diagram for cross-deploying to DEV, STAGING, and PROD- accounts.

As depicted in the diagram, the Build - the stage is now shorter. You learned that the following stages can reuse the artifact resulting from it. Yes, that makes sense.

You go further, and your thought is CDK as Infrastructure as Code shines because it is written in Typescript, allowing object-oriented programming paradigms to be applied. It is time to get your hands dirty, and you put the “Simple Pipeline” idea into an abstract class.

/**
 * Abstract Class for building multiple Pipelines
 * Make it abstract to give the Pipeline a base
 */
export abstract class PipelineStack extends Stack {
  public readonly pipeline: Pipeline;
  public readonly artifact: Artifact;

  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    // 0. Stage: CodePipeline (Pre-Requisite)
    // 0.1 Create the pipeline
    this.pipeline = new Pipeline(this, "Pipeline", {
      pipelineName: id,
    });

    // 0.2 Create the artifact to store outputs of the pipeline
    this.artifact = new Artifact("SourceArtifact");

    // 1. Stage : CodeCommit
    // We are importing the repository because it's a stateful resource
    // and should be deployed in a stateful stack
    const repository = Repository.fromRepositoryName(
      this,
      "Repository",
      "aws-cdk-pipeline-demo",
    );

    // 1.2. Add the source action to the pipeline
    this.pipeline.addStage({
      stageName: "Source",
      actions: [
        new CodeCommitSourceAction({
          actionName: "Source",
          output: this.artifact,
          repository,
          branch: props.branch ?? "main",
        }),
      ],
    });
    // 2. Stage: CodeBuild
    // 2.1. Build Stage
    this.pipeline.addStage({
      stageName: "Build",
      // We can actually paste many actions in here
      // But since we can dump anything in one CodeBuild
      actions: [
        new CodeBuildAction({
          actionName: "Build",
          input: this.artifact,
          project: new PipelineProject(this, "BuildProject", {
            projectName: "aws-cdk-build-project-demo",
            // You can also create your own buildspec.yml file and reference it here instead of using the inline buildspec
            // https://docs.aws.amazon.com/codebuild/latest/userguide/build-spec-ref.html#build-spec-ref-syntax
            buildSpec: BuildSpec.fromObject({
              version: "0.2",
              phases: {
                install: {
                  commands: ["npm install"],
                },
                build: {
                  commands: [
                    "npm run lint",
                    "npm run test:unit",
                    "npm run build",
                  ],
                },
              },
              artifacts: {
                "base-directory": "dist",
                files: ["**/*"],
              },
            }),
          }),
        }),
      ],
    });

    // 2.2. Deploy Stage
    // This part is now handled in the Concrete class which inherits this
    // However, we can simplify the inheritance and provide helper functions below
    // createPipelineProject(), addPoliciesToProject(), addStageToPipeline(), createPipelineProject(), addStageToPipeline()
  }

  /**
   * The function creates a pipeline project with specified commands and build specifications.
   * @param {Construct} scope - The scope parameter is the parent construct that the pipeline project
   * will be created under. It defines the scope or context in which the project will exist.
   * @param {string} name - The name of the pipeline project. It is used as the project name and also as
   * the name of the construct in the AWS CDK.
   * @param {Command} commands - The `commands` parameter is an object that contains the commands to be
   * executed in different phases of the build process. It has the following structure:
   * @returns a new instance of the `PipelineProject` class.
   */
  public createPipelineProject(
    scope: Construct,
    name: string,
    commands: Command,
  ): PipelineProject {
    return new PipelineProject(scope, name, {
      projectName: name,
      environment: {
        buildImage: LinuxBuildImage.STANDARD_7_0,
        computeType: ComputeType.LARGE,
      },
      buildSpec: BuildSpec.fromObject({
        version: "0.2",
        phases: {
          pre_build: {
            commands: commands.preBuild,
          },
          install: {
            commands: commands.install,
          },
          build: {
            commands: commands.build,
          },
          post_build: {
            commands: commands.postBuild,
          },
        },
        artifacts: {
          "base-directory": ".",
          files: ["**/*"],
        },
      }),
    });
  }

  /**
   * The function adds a stage to a pipeline with a given stage name and a list of actions.
   * @param {string} stageName - A string representing the name of the stage to be added to the
   * pipeline.
   * @param {Action[]} actions - The `actions` parameter is an array of `Action` objects. Each `Action`
   * object represents a specific action or task that needs to be performed in the pipeline stage.
   */
  public addStageToPipeline(stageName: string, actions: Action[]): void {
    this.pipeline.addStage({
      stageName,
      actions,
    });
  }
}

The beauty of the code is that you can simplify methods for the concrete class. You create a method called createPipelineProject which takes a third parameter commands from the type Command :

interface Command {
  preBuild?: string[];
  install?: string[];
  build?: string[];
  postBuild?: string[];
}

The concrete class would use this method whenever it creates a CodeBuild-Action like

this.createPipelineProject(this, `Deploy-${account.stage}`, {
  install: ["./scripts/assume-role.sh"],
  build: [`npx cdk deploy --stage ${account.stage}`],
  postBuild: ["npm run test:integration"],
});

Wow, I simplified the method because it’s now abstracted from the parent, you say. Now, you implement CrossAccountPipelineStack

interface Account {
  stage: string;
  number: string;
  region: string; // or to narrow it:  "eu-central-1" | "us-east-1"
}

const accounts: Account[] = [
  {
    number: "111111111111",
    region: "eu-central-1",
    stage: "dev",
  },
  {
    number: "222222222222",
    region: "eu-central-1",
    stage: "staging",
  },
  {
    number: "333333333333",
    region: "eu-central-1",
    stage: "prod",
  },
];

export class CrossAccountPipelineStack extends PipelineStack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    for (const account of accounts) {
      // Create CodeBuild Project
      const deploy = this.createPipelineProject(
        this,
        `DeployTo${account.stage}`,
        {
          install: ["./scripts/assume-role.sh"],
          build: [
            `npx sst deploy --stage ${account.stage}`,
          ],
          postBuild: ["npm run test:integration"],
        },
      );

      // Create Action for Codepipeline
      const actions = [
        new CodeBuildAction({
          actionName: `Deploy-${account.stage.toUpperCase()}`,
          input: this.artifact,
          project: deploy,
          runOrder: 2,
        }),
      ];

      // Add Action to Pipeline
      this.addStageToPipeline(
        `Deploy-${account.stage.toUpperCase()}`,
        account.stage !== "dev"
          ? [
              new ManualApprovalAction({
                actionName: "ManualApproval",
                additionalInformation: `Review Before Deploy ${account.stage}`,
                runOrder: 1,
              }),
              ...actions,
            ]
          : actions,
      );
    }
  }
}

The CrossAccountPipelineStack will get the Source and the Build- stage out-of-the-box, and the following is a loop through accounts. You can even add an action as you did in line 56, whether we want to add a ManualApprovalAction to our stage.

You wonder now how you can add that to your Monorepo project.

Pipeline in a Monorepo setup

Your project has a backend, and a frontend inside the packages.

Here is where it can get complex.

When to trigger the pipeline? Will all packages be deployed? How can I only deploy the packages related to the files which have been changed?

You think you can only solve this with “Multiple Pipelines”.

Multiple Pipelines

Okay, you think that each package needs to create its own CodePipeline. Since the setup is the same, you copy the CodePipeline from the diagram above.

But when should the pipeline be triggered? After researching, you find out that almost all services emit events. Thus, you can use EventBridge with Lambda. You are going to note that down.

AWS emits an event every time something changes in our CodeCommit. We can catch that with EventBridge. The event pattern for the event looks like

{
  "detail-type": ["CodeCommit Repository State Change"],
  "resources": ["repository.repositoryArn"],
  "source": ["aws.codecommit"],
  "detail": {
    "referenceType": ["branch"],
    "event": ["referenceCreated","referenceUpdated"],
    "referenceName": ["main"]
  }
}

You add that in your abstract class PipelineStack and create the respective Lambda and a policy for triggering CodePipeline to check for differences in CodeCommit.

// MultiPipelineStack: Lambda to trigger the correct Pipeline
// Because it's event-based, we need to understand the event pattern.
const eventPattern = {
  "detail-type": ["CodeCommit Repository State Change"],
  resources: [repository.repositoryArn],
  source: ["aws.codecommit"],
  detail: {
    referenceType: ["branch"],
    event: ["referenceCreated", "referenceUpdated"],
    referenceName: [props.branch ?? "main"],
  },
};

// Policy for the Lambda
const initialPolicy = [
  new PolicyStatement({
    actions: ["codecommit:GetDifferences"],
    resources: [repository.repositoryArn],
  }),
  new PolicyStatement({
    actions: ["codepipeline:StartPipelineExecution"],
    resources: [`arn:aws:codepipeline:${this.region}:${this.account}:*`],
  }),
];

const customLambdaPipelineTrigger =
  // We need to check if the function already exists
  // Otherwise whenever we create a new pipeline it would create a new Lambda but we only need one
  Function.fromFunctionName(
    this,
    "CustomLambdaPipelineTrigger",
    "customLambdaPipelineTrigger",
  ) ??
  new Function(this, "CustomLambdaPipelineTrigger", {
    handler: "packages/core/src/customLambdaPipelineTrigger.handler",
    description:
      "Trigger the pipeline when a commit is pushed to the master branch",
    functionName: "customLambdaPipelineTrigger",
    initialPolicy,
  });

// This emits an event and will halt before the pipeline is triggered
this.pipeline.onEvent("EventTrigger", {
  eventPattern,
  description: "Trigger the pipeline when a commit is pushed to the branch",
  target: new LambdaFunction(customLambdaPipelineTrigger),
});

You realize that you need to check if the Lambda Function already exists; otherwise, whenever a new pipeline is created, it will create a new Lambda.

Now, the pipeline will not be triggered immediately. Instead, the event is sent to EventBridge, which forwards it to Lambda. Lambda decides when to start which pipeline based on the logic.

The CustomLambdaPipelineTrigger- Lambda

Okay, now you need to write a Lambda and decide to go with Typescript. AWS CDK comes with an interface EventPattern which it can extend from.

import type { EventPattern } from 'aws-cdk-lib/aws-events';

interface CodeCommitStateChangeEvent extends EventPattern {
  detail: {
    callerUserArn: string;
    commitId: string;
    oldCommitId: string;
    event: string;
    referenceFullName: string;
    referenceName: string;
    referenceType: string;
    repositoryId: string;
    repositoryName: string;
  };
}

Because it will compare the file differences and execute the CodePipeline, you add the @aws-sdk/client-codecommit and @aws-sdk/client-codepipeline npm packages.

import { CodeCommitClient } from '@aws-sdk/client-codecommit';
import { CodePipelineClient } from '@aws-sdk/client-codepipeline';

const codecommitClient = new CodeCommitClient({});
const codepipelineClient = new CodePipelineClient({});

This is the structure of your Monorepo

├── packages
│   ├── backend
│   ├── core
│   └── frontend
├── stacks
│   ├── BackendStack.ts
│   ├── FrontendStack.ts
│   └── PipelineStack

The core is a shared package between backend and frontend. In total, you need two pipelines. Furthermore, you define the paths for the respective pipeline.

const stacksPath = "stacks";
const packages = "packages";
const commonFolder = ["stacks/PipelineStack", `${packages}/core`];

const PipelinePath = {
  backend: [
    `${packages}/backend`,
    `${stacksPath}/BackendStack.ts`,
    ...commonFolder,
  ],
  frontend: [
    `${packages}/frontend`,
    `${stacksPath}/FrontendStack.ts`,
    ...commonFolder,
  ],
};

const Pipeline = {
  frontend: "FrontendStackPipeline",
  backend: "BackendStackPipeline",
};

If something is changed in the commonFolder, each pipeline is triggered. Otherwise, it depends on the packages - path. It is essential to put the correct name in the Pipeline- object.

After you set up the pre-requisite, you write the Lambda handler

export async function handler(event: Prettify<CodeCommitStateChangeEvent>) {
  // Use the SDK to get the difference from the new commit and previous commit
  const getDifferences = new GetDifferencesCommand({
    repositoryName: event.detail.repositoryName,
    afterCommitSpecifier: event.detail.commitId,
    beforeCommitSpecifier: event.detail.oldCommitId,
  });
  const codecommit = await codecommitClient.send(getDifferences);

  // iterate over the paths in PipelinePath to check which pipeline should be triggered
  // e.g. if the path includes functions/src/Application, then trigger the AppPipeline
  for (const path in PipelinePath) {
    const typePath = path as keyof typeof Pipeline;
    const pipeline = Pipeline[typePath];

    if (codecommit.differences) {
      const check = codecommit.differences.some((difference) => {
        return PipelinePath[typePath].some((substring) => {
          // Check if the path includes the substring
          return difference.afterBlob?.path?.includes(substring);
        });
      });

      if (check) {
        const triggerPipelineCommand = new StartPipelineExecutionCommand({
          name: pipeline,
        });
        await codepipelineClient.send(triggerPipelineCommand);
      }
    }
  }
}

1. It gets the differences between two commits with GetDifferencesCommand (Line 3-8).

2. The for-in- The loop goes through each file path of the packages (Lines 12-31).

3. It extracts the pipeline type path and gets the path according to the pipeline name (Lines 13-14).

4. If there are differences resulting from the GetDifferencesCommand , it uses the some method tests whether at least one element in the array passes the test implemented by the provided function. In this case, it's checking if any of the differences have a afterBlob.path that includes any of the substrings in PipelinePath[typePath] (Line 16-22)

5. If Step 4 is true, it triggers the pipeline according to the pipeline by using the SDK (Lines 24-29)

MultiPipelineStack

Along the way, you find an excellent framework, SST. Which is much faster than CDK and provides a better DX^1. Here is how you then define your MultiPipelineStack.

import { StackContext } from "sst/constructs";
import { CrossAccountPipelineStack } from "./CrossAccountPipelineStack";
import { accounts } from "./accounts";

export function MultiPipelineStack({ stack }: StackContext) {
  new CrossAccountPipelineStack(stack, "FrontendPipelineStack", {
    accounts,
    purpose: "frontend",
  });

  new CrossAccountPipelineStack(stack, "BackendPipelineStack", {
    accounts,
    purpose: "backend",
  });
}

Since you have defined a CrossAccountPipelineStack you put multiple of them in that stack. You distinguish them by adding a purpose into the StackProps, which you then renamed to CrossAccountPipelineStackProps. This is needed to add it to the environment variable. PURPOSE.

// in CrossAccountPipelineStack.ts
type CrossAccountPipelineStackProps = PipelineStackProps & {
  purpose: string;
};

export class CrossAccountPipelineStack extends PipelineStack {
  constructor(scope: Construct, id: string, props: CrossAccountPipelineStackProps) {
                                                    // ^ This is changed
    for (const account of accounts) {
      // Create CodeBuild Project
      const deploy = this.createPipelineProject(this, `DeployTo${account.stage}`, {
        install: ['./scripts/assume-role.sh'],
        // This is how you create a new environment variable
        build: [`PURPOSE=${props.purpose} npx sst deploy --stage ${account.stage}`],
        postBuild: ['npm run test:integration'],
      });
      // ...
    }
  }
}

The deployment is controlled by the PURPOSE environmental variable inside the sst.config.ts .

import { SSTConfig } from "sst";
import { BackendStack } from "./stacks/BackendStack";
import { FrontendStack } from "./stacks/FrontendStack";

export default {
  config(_input) {
    return {
      name: "aws-ug-codepipeline-demo",
      region: "eu-central-1",
    };
  },
  stacks(app) {
    switch (process.env.PURPOSE) {
      case "backend":
        app.stack(BackendStack);
        break;
      case "frontend":
        app.stack(FrontendStack);
        break;
      default:
        throw new Error("PURPOSE environment variable is not set");
    }
  },
} satisfies SSTConfig;

Conclusion

Puh, you have written a lot of code, but you’re satisfied with the work and could even impress the other Engineers, especially your CTO.

You are proud that you have leveraged AWS CDK to make your CodePipeline less suck. You enabled the OOP paradigm to reuse code and abstract classes for the concrete class to have a pipeline foundation. It can fill the need for cross-account deployment and trigger the correct pipeline when having multiple pipelines in a Monorepo project.

For now, the backend team can push to the main- branch, and it triggers the BackendPipeline and the FrontendPipeline will deploy the NextJS application whenever a new file changes in the frontend- package.

However, you know there are ways to improve the pipeline by adding notifications and caches for CodeBuild and triggering a pipeline whenever a Pull Request has been created.

Another powerful asset is L3-Constructs or patterns. L3- constructs simplify multiple resources for common tasks. One of them is the ECS-Pattern.

However, they abstract away many resources, and in this blog post, I am going to demystify the ApplicationLoadBalancedFargateService construct.

ApplicationLoadBalancedFargateService

Let’s break down the verbose construct ApplicationLoadBalancedFargateService into their resources and what code you may need.

Minimal

Below is the minimal code that deploys an Application Load Balancer, a Fargate ECS Cluster and task, Target Groups, and a Listener on port 80. Depending on the deployment region, the code deploys a VPC and at least two private and public subnets.

const service = new ApplicationLoadBalancedFargateService(stack, "AlbFargateService", {
    // You have to define a task image option
    taskImageOptions: {
      image: ContainerImage.fromAsset("path/to/your/Dockerfile"),
    },
  },
);

Or if you want to use your own taskDefinition:

const service = new ApplicationLoadBalancedFargateService(stack, 'AlbFargateService', {
  // The task definition will mostlikely be bigger as you need to pass props
  taskDefinition: new FargateTaskDefinition(stack, 'TaskDefinition', {
    // here you will need add props which looks similar to above `taskImageOptions`
  })
})

That is what you will get. Out of simplicity, I left out the IAM roles and policies.

When using the taskDefinitionyou often need to configure the container, which usually ends up in more code.

HTTPS

However, the minimal code approach only listens on port 80 (HTTP). If you want to create a serious application using AWS ECS Fargate, you want HTTPS (port 443).

As a pre-requisite, you must have a Hosted Zone and a domain name in your Route53.

new ApplicationLoadBalancedFargateService(stack, "AlbFargateServiceHttps", {
  taskImageOptions: {
    image: ContainerImage.fromAsset("path/to/your/Dockerfile"),
  },
  publicLoadBalancer: true,
  redirectHTTP: true, // Best practice to redirect if calling URL with `http`
  cpu: 512, // <-- Default: 0.25 Otherwise container dies to quickly
  memoryLimitMiB: 1024, // <-- Otherwise container dies to quickly
  domainName: "my.example.com",
  domainZone: HostedZone.fromHostedZoneAttributes(this, "HostedZone", {
    hostedZoneId: props.hostedZoneId,
    zoneName: props.zoneName,
  }),
});

And this is what it looks like

7 extra lines create a second listener on port 443, and the Application Load Balancer (ALB) terminates HTTPS with an AWS Certificate Manager (ACM) certificate. This certificate will be automatically created when adding domainName and domainZone.

Now, you will be able to access your container with https://my.example.com.

Authentication with Cognito

Cognito is a great option if we want to add an authentication layer. This means we must authenticate with Cognito before loading data into the Docker container. To achieve this, we add an action to our listener that tells the user to authenticate before being directed to the container.

// Define the Service first
const service = new ApplicationLoadBalancedFargateService(
  stack,
  "AlbFargateService",
  {
    taskImageOptions: {
      image: ContainerImage.fromAsset("path/to/your/Dockerfile"),
    },
    publicLoadBalancer: true,
    redirectHTTP: true, // Best practice to redirect if calling URL with `http`
    cpu: 512, // <-- Default: 0.25 Otherwise container dies to quickly
    memoryLimitMiB: 1024, // <-- Otherwise container dies to quickly
    domainName: "my.example.com",
    domainZone: HostedZone.fromHostedZoneAttributes(this, "HostedZone", {
      hostedZoneId: props.hostedZoneId,
      zoneName: props.zoneName,
    }),
  },
);

// Set Cognito
const userPool = new UserPool(this, "UserPool");
const userPoolClient = new UserPoolClient(this, "UserPoolClient", {
  userPool,
  generateSecret: true,
  authFlows: {
    userPassword: true,
  },
  oAuth: {
    flows: {
      authorizationCodeGrant: true,
    },
    callbackUrls: [
      `https://${service.loadBalancer.loadBalancerDnsName}/oauth2/idpresponse`,
    ],
  },
});
const userPoolDomain = new UserPoolDomain(this, "Domain", {
  userPool,
  cognitoDomain: {
    domainPrefix: "test-cdk-prefix",
  },
});

// Add actions to Target Groups
service.listener.addAction("CognitoListener", {
  action: new AuthenticateCognitoAction({
    userPool,
    userPoolClient,
    userPoolDomain,
    next: ListenerAction.forward([service.targetGroup]),
  }),
});

Line 34 is crucial for redirecting back to the load balancer domain. Lines 46 - 53 describe the new action for the listener, where we add the Cognito Suite from lines 22-43.

The “simplified” diagram

Cognito with Custom Domain and Identity Federation

If we want to use an Identity Federation like GitHub, we can utilize an L2-Construct UserPoolIdentityProviderOidc. Moreover, if we aim to establish a personalized domain for our login page, we need to generate a certificate initially and then add it to Route 53's Hosted Zone.

declare const service: ApplicationLoadBalancedFargateService // like in the previous code
declare const userPool: UserPool(this, 'UserPool') // like in the previous code
declare const userPoolClient: UserPoolClient(this, 'UserPoolClient') // like in the previous code
const userPoolIdentityProviderOidc = new UserPoolIdentityProviderOidc(this, 'GitHubOidc', {
    clientId: 'register-the-app-on-github',
    clientSecret: 'get-it-on-github',
    issuerUrl: 'https://github.com',
    userPool,
    name: 'GitHub',
});

const hostedZone = HostedZone.fromHostedZoneAttributes(this, 'HostedZone', {
    hostedZoneId: 'my-id',
    zoneName: 'example.com',
})
const certificate = new CdkCertificate(this, 'Certificate', {
    domainName: 'github.example.com',
    validation: CertificateValidation.fromDns(hostedZone),
}); // ❗ You need a new one
const userPoolDomain = new UserPoolDomain(this, 'UserPoolDomain', {
    userPool,
    customDomain: {
        certificate,
        domainName: 'github.example.com',
    }
});

service.listener.addAction('CognitoListener', {
    action: new AuthenticateCognitoAction({
        userPool,
        userPoolClient,
        userPoolDomain,
        next: ListenerAction.forward([service.targetGroup]),
    }),
});

Note: Lines 1-3 have the same implementation as in the previous section.

It is possible to use Cognito with Identity Federation and a custom domain. However, you may not require a Cognito Custom Domain if you already use ID Federation with services like GitHub, Google, or Azure AD. A custom domain is available for completeness but may not be necessary in certain cases.

Conclusion

The ECS pattern is very powerful and provides many services with less code. They still provide properties for customization. However, you most likely end up with more code (at least because of having HTTPs). What we haven’t seen are roles and policies. Do not worry the community won’t let you down and will try to reach the least privileged. And even if you have doubts, you could easily add your roles or policies.

In this blog post, we have demystified the ApplicationLoadBalancedFargateService pattern in their services, and I hope you understand this pattern a bit better. I recommend using them. However, don’t blindly trust them; check their underlying resources.

When ESLint announced that they deprecated their formatting rules, my reaction was “WTF”… I am aware that the ESLint team has always been advised against using them as a formatter. However, I disagree.

Why I think you should use it as a Formatter

Antfu made good points about why he does not use Prettier.

“It’s a Mess with ESLint” [^1]

And I agree. I spent so much time configuring both together…

https://x.com/antfu7/status/1279149212974776320?s=20

For instance, my auto-fix settings in VS Code for ESLint and Prettier would constantly overwrite each other, causing frustration and wasting time.

When working with ESLint and Prettier, it's common to end up with two configuration files (and possibly two ignore files, making it a total of four files). One configuration file is for ESLint, and the other is for Prettier. I would advise against putting the configuration in the package.json file as it would make it unnecessarily large, and you would be limited to using JSON format. It would require additional effort to maintain.

Linters are tools that verify and correct logical and non-whitespace style issues in code, such as naming consistency and bug detection. Linters often take seconds or more to run because they apply many logical rules to code. [²]

I think if you configure your VS Code correctly, it does lint on the fly. Why shouldn’t it detect stylistic rules? Yes, it may take a few seconds to run eslint . —fix , but I don’t think it is slow. Of course, it depends on the project size. But would it be much faster when we put the stylistic rules into Prettier and then run prettier . —write. Which one should be executed first, ESLint or Prettier? 😵‍💫

Take advantage of the powerful capabilities of your IDE and let it handle linting and formatting on the fly. Here is my VS Code setting:

{
  "editor.formatOnSave": true,
  "typescript.tsdk": "./node_modules/typescript/lib",
  "eslint.format.enable": true,
  "editor.codeActionsOnSave": {
    "source.fixAll.eslint": true
  },
  "typescript.format.enable": false,
  "prettier.enable": false
}

I disabled all formatter except for ESLint to make sure only ESLint is used here.

It’s very opinionated [³]

Prettier forces you to use printwidth. Of course, you can adjust it. However, most likely, you have already run the formatter… Grr

See below and build your own opinion if Prettier makes this really “prettier” 🫠

Okay, they deprecated it. What should we do now?

Using an older version is possible, but it is not recommended.

Anthony Fu started the project ESLint Stylistic, an ESLint plugin with all the deprecated formatting rules. The maintainer is one of the best, and I am pretty sure he will come up with something great.

But if you're looking for a reliable and efficient tool to manage your code, Biome is the way to go. In my experience, it's far more effective than ESLint. I highly recommend trying it and seeing the difference it can make in your workflow, and you get a linter and a formatter in one tool.

And their benchmark is very promising^4:

• Biome's ~25 times faster than Prettier

• Biome's ~20 times faster than parallel-prettier

• Biome's ~20 times faster than xargs -P

• Biome's 1.5-2 times faster than dprint

• Biome single-threaded is ~7 times faster than Prettier.

Outlook

There is an interesting competition on going to rewrite the prettier in Rust. I can only guess that Prettier will get faster.

Interestingly, the Biome team is also joining that competition. https://twitter.com/biomejs/status/1726365565348991346 I can imagine they will be merged together.

Conclusion

ESLint has deprecated its formatting rules. The ESLint team advises using a Formatter just formatting. However, if your Linter is already reading your codebase, why should you run another tool to reread it? Luckily, there are several alternatives available that you can consider.

I advise using one tool for lining and formatting your code, whether using Biome or ESLint with the Stylistic Rules plugin, but that would avoid some maintenance overhead. Furthermore, if you have configured your IDE, it can lint and format your code on the fly.

And if you don’t work on Google’s Monorepo, I think you wait for the time to finish formatting 😉

[²]: https://typescript-eslint.io/linting/troubleshooting/formatting/
[³]: https://eslint.style/guide/why

“October, crisp, misty, golden October, when the light is sweet and heavy.” ― Angela Carter, The Magic Toyshop

When the leaves turn their color, and the sun is enjoyable, ahh, October, you beautiful month. When the days are shorter, and you don't feel bad when it's raining, oh yes, it's Hacktober.

How has my #Hacktoberfest?

Even though I haven't got all my PRs approved, I can say it was very good.

I am a fan of open source, but honestly, I don't contribute much. Hacktoberfest motivates me to change that.

I use @OpenSauced to search for repositories that have the #hacktoberfest label. It's a fantastic tool for finding excellent repositories to contribute to. I understand that it may not capture all available repositories, but the goal of Hacktoberfest is to discover something exciting and make a meaningful contribution to it.

And I found so many great projects.

• trigger.dev--) for running background jobs framework with Typescript
• Steampipe--) Using SQL to query your cloud services without a database
• SAML jackson (Best(Best) name!!)- for a seamless integration of a single authentication
• Novu--) The notification center infrastructure for your app

and many more. Go check it out yourself here.

My contributions

Komiser

I often work with AWS and recently came across a tool called Komiser, which can create an inventory of your cloud assets and display the costs of each cloud service. I found it quite interesting, but I noticed a bug where services were labeled incorrectly. Since the tool is written in Golang, I tried to fix the bug myself, but someone else came up with a better solution. Although it was my first time working with Go, Phind taught me about goroutines, which are used in Go's concurrent model. I was eager to contribute to the repository, so I added a new pricing support for CloudFront Functions. Additionally, I could apply my Javascript knowledge to animate the edges of their explorer.

My learnings:

• Usage of Go and their concurrent execution.
• How the pricing of CloudFront Functions are calculated, and that they differ from Lambda@Edge
• Refreshing my NextJS skills and how to use Cytoscape Layer

DocsGPT

I became interested in DocsGPT because I had a similar idea of training a model based on my own documents. I started reading through their roadmap and engaging in their Discord channel. One of the ideas was to put Langchain's Vectorstore on AWS S3 or connect the application with an AWS S3 storage. DocsGPT is written in Python, and I found it to be a good project to brush up on my Python skills. The pull request is currently being worked on.

My Learnings here:

• Use LangChain to develop a language model
• Get to improve my rusty Python skills

S3-Simpler

I realized that I had already started working on my own project called S3-Simpler (S4) for last year’s #hacktoberfest. The main aim of this project is to create a small wrapper that can be used for uploading, downloading, and sharing any type of file, regardless of its size. Additionally, this repository also includes a CLI that can be run to upload, download, and share your files of an S3-Bucket (I was thinking of running something like npx s4 upload --bucket my-bucket --filepath /path/to/my/file). Typically, multipart uploads are required for files larger than 1 GB, but this CLI simplifies the process.

This project is built using NodeJS and has a Monorepo setup. I attempted to use Bun for bundling, but it had issues with bundling CommonJS packages to ESM. The library uses AWS SDK JS v3, and the CLI uses ink. It's still under development and needs to be published to npm. Any feedback and contributions would be greatly appreciated.

However, this project was forgotten, and #Hacktoberfest is over. Despite having a label, there was no marketing done, and as a result, it wasn't as visible as it could have been.

My Learnings:

• Without any marketing, you won't get any visibility
• Project Management is harder than I thought
• Trying out Bun and realized its bundler was not correctly bundling to ESM
• Creating a library for ESM and CJS (Dual-Mode)

Conclusion

Throughout Hacktoberfest, I attempted to complete four pull requests. Nonetheless, the experience was a great opportunity for me to learn and grow. I participated in various projects, including those related to AI, my own personal projects, and tools that I use for my daily work. This allowed me to gain exposure to a wide range of different technologies and to learn new skills that I can apply to future projects.

Moreover, the event provided me with the chance to explore many great projects, which helped me to expand my knowledge and develop my coding abilities. Beyond the hacking and coding aspects, I also enjoyed the social aspect of the event, interacting with others on Discord channels and getting to know new people who share my passion for open source.

Hacktoberfest motivated me to commit to doing something every day. Although I wasn't able to complete all four PRs, I am eager to continue contributing to open-source projects, as I believe that this is an important way to give back to the community and to help others. I am grateful for the opportunity to have participated in Hacktoberfest and look forward to continuing to learn and grow as a developer.

Now, it’s most likely that you burn yourself or you are smart and let it cool down a bit.

Bun is so cool that I also got hyped. However, I questioned myself:

• How can I use it in my current or future projects?

• How can I deploy it?

• Do I have to rewrite a lot or even everything?

In this blog post, we will discover the usage of Bun by creating an AWS CDK template using Bun tooling.

What is Bun?

The NodeJS was established long ago but had no unified tooling and an ever-changing ecosystem. Bun wants to become the all-in-one JavaScript toolkit, including runtime, package manager, test runner, and bundler. It is built from scratch with three primary design goals: fast startup, fast running performance, and cohesive DX.

What does it mean “Runtime”?

It means how your computer interprets your script. Each programming language has its Syntax and grammar; you can run a program using their language^1. For example, if you write a Python script, you run it with python myscript.py . The machine uses Python as runtime to interpret it. The file extension allows humans to understand that this is a Python file or editors to do color highlighting, but it’s unnecessary for the machine. The same applies to NodeJS and Bun. You can execute the script by running node myscript.js and bun myscript.js. Notice that Bun can run Javascript.

More interesting is what Syntax Bun supports. - A bun of (pun intended ✊) including Typescript, JSX, and JSON².

What if we want to deploy a function to any Cloud provider such as Cloudflare or AWS? None of them support these for now (28. September 2023).

Remember, we need to run bun myscript.ts . Luckily, on AWS Lambda, we could create a custom runtime.

That means we don’t need to transpile the Typescript code to ESM or CJS. Currently, only Deno Deploy can run your Typescript function out of the box. However, in order to keep the code small, we still need some sort of bundling. Luckily, Bun is also a bundler 😉

What is a Bundler?

A bundler is a tool that puts your code into a single file. When you develop a function, you may split your code into multiple files and use third-party packages. So, instead of shipping the whole node_modules- folder or all the libraries, a bundler only uses the correct code.

We do this to optimize JavaScript files by reducing their size and number, which can significantly improve website performance, resulting in better user experience and increased engagement.

Furthermore, Node-Server cannot understand Typescript, so we use a bundler to transpile and bundle.

The famous bundler is Webpack, but esbuild is the fastest one… Well, until Bun 🍔

Bunnyfied AWS CDK project 🐰

As a cloud engineer, I am working a lot with AWS CDK, a tool for infrastructure on AWS. Usually, I develop my AWS Resources in SST ([here are my thoughts on SST](here my thoughts on SST) and not in Projen ([here are my thoughts on Projen](here my thoughts on SST).

AWS CDK

AWS CDK is an open-source software development framework developed by Amazon Web Services (AWS) that allows developers to define and provision cloud infrastructure resources using familiar programming languages.

Their CLI bootstraps an npm project. I used that to Bunyfied it.

bunx cdk init app --language typescript

For fun, I decided to test which package manager is the fastest for creating an AWS CDK project. It appears that bunx is faster than npx and pnpx. However, this was not a proper benchmarking test.

On this attempt bunx runs the fastest but is not really impressive.

bun test and bun install

When I first did it, I removed jest and package-lock.json because I wanted to try out the native Bun Test, and I used Bun’s package manager to install packages.

Testing

Bun comes with its own testing suite, and it gives you nice hints in order to make this test pass 😇

And I have to say the --watch mode is really fast. vitest is also fast and reruns only failed tests, which makes it, in the end, faster (here when using Monorepo).

However, Mocking didn't work as expected. I attempted to spy on a function to manipulate a result but was unsuccessful. Their mocks provided were also unusable when trying to mock out the functions of packages. The documentation was not helpful either. As a result, I had to abandon bun test and used vitest instead.

Package Manager

Bun is clearly the fastest, and this time, you can really feel it 🤩

When I installed Lambda Powertools, Bun became significantly faster. It turns out that I had already installed some packages, but still, Bun was faster than pnpm. 

Workspaces

I created a Monorepo because Bun supports Workspaces).

Deploying a Bun Runtime

Bun has its own package for creating a Lambda Layer.

I created a BunFunLayerStack to deploy a Lambda Layer. That layer is customizable and could be consumed by another account, or you could even make it public.

BunFun Construct

Then, I created a Construct BunFun which creates a Function using the Lambda Layer.

It tries to read Arn, resulting from the BunFunLayerStack but can be overwritten when passing props.bunLayer in case you bring your own Layer.

By default, the respective target will be used. If it's a Typescript or Javascript file, then node it will be the target. But then you could use the NodeJsFunction- Construct of CDK. However, when you need the Bun there is a bunConfig- property that also opens up all the Bun configurations from their bundler API.

I configured that it is optional, but when you use it, make sure you add a target. Sometimes I love Typescript 🤩

Generally, I put three examples in the BunFunStack.

Run it

I put a couple of Lambdas in the packages/functions to get started and I realized that the Lambdas using that runtime must always return a new Response. Otherwise, the Lambda will not execute successfully. But that doesn't mean you cannot use it for handling SQS.

BunFunStack and BunFunLayerStack can be deployed separately. In this repo, I put them into an BunOvenStack.

Simply run bun deploy:all to deploy the BunOvenStack and run bun deploy:only to deploy only the BunFunStack. The bun deploy:only is also parameterized and could run another stack e.g. STACK=BunFunLayerStack bun deploy:only.

Conclusion

In this blog post, I used Bun's tooling to create an AWS CDK project. It includes a Layer for deploying a Bun Runtime and a Construct for creating a BunFunction.

Why should we care? Well, Bun would eliminate the fact of either using CommonJS or EcmaScript Modules (ESM) because most of the developers (me included) simply don't care. With the Bun runtime, you would run your Typescript files without transpiling them. Furthermore, Bun is fast. This blog does not cover benchmarking the Lambdas with a Bun runtime as there are already Benchmarks for Bun’s performance[^4](%5Bhttps://medium.com/@mitchellkossoris/serverless-bun-vs-node-benchmarking-on-aws-lambda-ecd4fe7c2fc2%5D(%3Chttps://medium.com/@mitchellkossoris/serverless-bun-vs-node-benchmarking-on-aws-lambda-ecd4fe7c2fc2%5D(https://medium.com/@mitchellkossoris/serverless-bun-vs-node-benchmarking-on-aws-lambda-ecd4fe7c2fc2)%3E)).

Unfortunately, bun test was not usable, and I use vitest instead. Moreover, the Lambda Runtime requires you always return new Response when using the bun target. This could be confusing when using Lambda for a different purpose then using an API Gateway.

With my Bunnyfied AWS CDK template, you could already start deploying Lambda functions using Bun Runtime, so give it a shoot 😉 To try it out, use the following command bun create github.com/jolo-dev/bunnyfied-aws-cdk

P.S.: It's faster than the npx cdk init app. P.S.S: Feedback and Contributions are welcome!

Developers are increasingly adopting serverless computing since it removes the requirement to manage infrastructure. When it comes to creating Infrastructure as Code, AWS CDK has become a favored option. Nonetheless, the Developer Experience (DX) for creating serverless applications with CDK isn't optimal. For instance, the deployment time can be excessively long when building REST APIs and debugging Lambdas can become tedious.

The Serverless Stack (SST) framework simplifies the process of building serverless applications on AWS. This article explores SST's architecture, features, and benefits for building and deploying serverless applications.

What's wrong with Vanilla CDK?

AWS CDK (I call it vanilla CDK) generates Cloudformation, AWS native Infrastructure as Code, by using a high-level programming language such as Typescript, Python, or Go.

As someone who has worked on numerous CDK codebases, the deployment process and developer setup have often caused me delays. Though CDK hotswap has been a significant improvement, it still tends to be sluggish.

Developing your REST API with API Gateway and Lambdas in CDK is quite painful. AWS SAM and Serverless Framework offer local testing but often I believe you get better feedback when directly deploying on AWS. Debugging with CDK is not streamlined as you need to find the correlated logs. And we all know, the AWS Console sucks.

In my article, Strategies for a faster Lambda Development, I was mentioning SST.

Why SST?

SST offers powerful monitoring and debugging tools while allowing developers to focus on application logic. It comes with its own UI console based on your CDK stack.

Why should I care about its Console?

Picture this scenario: You have created a stack using several Lambdas and now you need to debug them. To do so, you head over to AWS Cloudwatch and try to locate their logs. Unfortunately, you realize that you have redeployed the stack and now you have to sift through numerous logs to locate the correct log file.

SST’s console shows their log files which makes debugging so much faster.

However, you won’t find all AWS resources but only those which are related to Serverless (Lambdas, S3, DynamoDB etc.). When utilizing a serverless approach, then it should be fine though 😉

Should I still consider developing it locally first?

With SST you don’t have to do it anymore as it (re)deploys super fast and it feels like local development. Really, try it yourself 🤯

It feels like that I do not wait for anything. Whereas in CDK’s hotswap, I have to wait until it’s deployed (you don’t see it but I was hitting refresh the whole time).

P.S.: I was a bit slow when changing the Lambda above, I am pretty sure SST is much faster.

Furthermore, SST allows live Lambda live developing which can be integrated into your IDE. That means you could set breakpoints and debug your application while it’s deployed on AWS.

Developer Experience (DX)

In general, the SST team put a lot of effort into the Developer Experience (DX). Their constructs are very intuitive. Here is an example with their API- construct.

const api = new Api(this, 'KeyVaultApi', {
  defaults: {
    function: {
      runtime: 'nodejs18.x',
    },
  },
  routes: {
    "GET /ping": "packages/functions/src/Foo/ping.handler",
    "GET /pong": "packages/functions/src/Foo/pong.handler",
  },
});

instead of

const pingLambda = new lambda.NodejsFunction(this, 'Ping', {
  functionName: 'ping',
  entry: 'src/Foo/ping.ts',
  handler: 'handler',
  runtime: Runtime.NODEJS_18_X,
  bundling: {
    minify: true,
  },
});

const pongLambda = new lambda.NodejsFunction(this, 'Pong', {
  functionName: 'pong',
  entry: 'src/Foo/pong.ts',
  handler: 'handler',
  runtime: Runtime.NODEJS_18_X,
  bundling: {
    minify: true,
  },
});

const pingPongApi = new apigateway.RestApi(this, 'ping-pong-api');
const ping = pingPongApi.root.addResource('ping');
ping.addMethod('GET', new apigateway.LambdaIntegration(pingLambda));
const pong = keyGenRestApi.root.addResource('pong');
pong.addMethod('GET', new apigateway.LambdaIntegration(pongLambda));

Cool, we reduced the lines of code by 14 and make the code more readable 😎.

Use vanilla CDK

SST, being built on top of CDK, allows for the integration of CDK constructs. This means that despite using SST, you still have the option to utilize the powerful features of CDK.

Documentation

I am always annoyed about CDK’s API reference documentation because there is no search and CMD + F does just half of its job. However, SST's implementation of Algolia's search feature makes it incredibly easy to locate not just their constructs, but anything relevant to your search term.

Wow, SST is amazing but does it have a downside?

Yep, unfortunately, there are some limitations.

Not possible to do cross-account deployments with CDK pipelines

We had already some CDK pipelines established and in fact, I have written an article about it. However, in the end, you have a CDK app that should be deployable via CDK pipelines… that’s what I thought.

I spent some time figuring out why it does not deploy to a different account. Assuming roles from a different account is not straightforward with CDK pipelines and it usually works with Vanilla CDK.

I found out that if you node_modules/sst/constructs/Stack.js manually remove Line 69 and Line 74 - 78, it

Stack.checkForPropsIsConstruct(id, props);
// Stack.checkForEnvInProps(id, props);

super(scope, stackId, {
  ...props,
//  env: {
//    account: app.account,
//    region: app.region,
//  },
  synthesizer: props?.synthesizer || Stack.buildSynthesizer(),
});

SST cannot be integrated into an existing CDK stack (but the other way round works)

I had problems integrating SST in my current CDK stack as if you bootstrapped your project with npx cdk init —lang typescript , you may get incompatibility between ESM and CommonJS as CDK is compiled to CommonJS (which is older) and SST uses ESM (which is a modern and modular approach).

UI is limited to Serverless Only services

SST console shows only Serverless service but no EC2 or ECS containers.

SST is not good for developing a library or constructs

If you want to create CDK library consisting of custom constructs, SST won’t be able to satisfy you. Which is fine as that’s not their intention.

However, if you intend to do so, you may read my thought about using Projen 🫣

Conclusion

If you're a developer, you'll appreciate how well SST caters to your needs. It provides a smooth and integrated solution for building full-stack applications and guarantees an exceptional developer experience.

The console provides a detailed overview of your serverless stack's associated resources and presents comprehensive outputs. No need to find anything on the AWS console 😜

Hopefully, this blog was helpful in showing you how to enhance your stack and take it to the next level. Consider using SST for your future AWS projects.

Still in doubt? Watch Fireship’s 100 seconds of SST.

Starting with AWS Lambda can be challenging, especially when optimizing and streamlining the development process. While running functions directly in the AWS Console is the simplest and quickest approach, it becomes tedious when adding packages, as it requires zipping and uploading them to the console – a method that is rarely used in practice.

An alternative workflow involves setting up Infrastructure as Code (IaC) tooling, such as AWS CDK and running deployment commands like sam deploy with AWS SAM. These tools handle code uploading and simplify the process.

Although using IaC tooling is a step forward, it can still be slow, as it often requires updating CloudFormation, which can take time. In this blog post, we will explore various strategies and tools to make Lambda development faster and more efficient.

Strategy: Bypassing Cloudformation

Many tools which are on top of Cloudformation (CFN) like CDK or AWS SAM, synthesize their code to a CFN- template before deploying to the cloud. That means they create a CFN template and make use of AWS native IaC-service. It is nothing wrong with that, it is just freaking slow. Let's have a look at the cdk deploy command

To begin, the command creates a CFN template and uploads it, along with a ZIP file, to the assets-bucket (which is created when cdk bootstrap is initiated, more information can be found here). The Cloudformation stack is then either created or updated, with drift detection being conducted to ensure any manual changes are accounted for. Finally, an AWS Lambda function is created alongside the stack to retrieve the ZIP-file.

We can automate many steps with this command, but updating Cloudformation for small changes takes a lot of time and can disrupt the development process. Let's explore cdk deploy --hotswap --watch, also known as cdk watch. This command uses --watch to detect any file changes and trigger updates automatically.

By using this command, we skip the CloudFormation step and directly call the Lambda-API. Additionally, the development experience is improved by running a watcher that detects file changes, giving the feeling of a hot module reload. However, it's important to note that this process may cause discrepancies between your CloudFormation template and the deployed application, and not all resource changes can be 'hot-swapped' (Ref. here). Therefore, it's recommended to only use this in development and use the cdk deploy command in production.

AWS SAM has also introduced a similar concept called SAM accelerate.

SST

The first time when I saw a 'hot swap' was at SST.

SST is a framework that makes it easy to build modern full-stack applications on AWS.

SST sits on top of CDK, and they had first the idea of bypassing Cloudformation. They call it Live Lambda Development.

Live Lambda Development or Live Lambda is feature of SST that allows you to debug and test your Lambda functions locally*, while being **invoked remotely by resources in AWS**. It works by proxying requests from your AWS account to your local machine.*

I am a big fan of SST because they do a much better job than plain CDK ❤️

• Much better documentation, (why the heck I cannot search in the CDK API doc?)

• Much better development experience (DX)

• Faster Development and Deployment

• Local UI locally for triggering the Lambda

Generally, if you consider developing Serverless Apps, you should consider SST. You won't be disappointed ;)

Strategy: Run Lambda locally

Another strategy could be to run an emulator locally which simulates your cloud provider. An emulator is a service with a bunch of APIs running locally in your container or as a web service.

Serverless Framework Offline

The Serverless Offline-plugin is the only local emulator which does not require Docker but creates a local HTTP Server in NodeJS (see the diagram above). It simulates an API gateway and simply invokes the lambda locally.

This only works with Lambda which is invoked via an API Gateway. This great but limited to API Gateway.

Nevertheless, there are a few plugins for Eventbridge, S3 Events, and DynamoDB local for local development which are sitting on top of Serverless Offline.

This is easy to set up and really fast and highly recommended when using Serverless Framework.

Localstack

Localstack is a cloud service emulator that runs in a single container on your laptop or in your CI environment.

It can be installed and run and make sure Docker is installed and also running

pip install localstack

# or in with docker
# docker run \
#   --rm -it \
#   -p 4566:4566 \
#   -p 4510-4559:4510-4559 \
#   localstack/localstack

localstack start -d # in Daemon mode (running in the background)

It offers a bunch of ports to run against it. But most of the core services are running on port 4566. You could now do all the AWS-CLI- commands against this endpoint by appending the flag --endpoint-url=http://localhost:4566. Or you can use a tool called awslocal. Localstack also comes with a lot of integrations for your development tools such as CDK, Terraform, AWS SAM and many more.

Let's have a look at cdklocal. I would recommend installing it not globally but in your project. pnpm install aws-cdk-local. Once you have created your CDK-stack, you could run npx cdklocal deploy. But whoops...

✨  Synthesis time: 4.89s

LambdaStack: building assets...


 ❌ Building assets failed: Error: Building Assets Failed: Error: LambdaStack: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)
    at buildAllStackAssets (/Users/jolo/Development/FTI/GeniusCloud/node_modules/aws-cdk/lib/index.js:383:115268)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async CdkToolkit.deploy (/Users/jolo/Development/FTI/GeniusCloud/node_modules/aws-cdk/lib/index.js:383:143485)
    at async exec4 (/Users/jolo/Development/FTI/GeniusCloud/node_modules/aws-cdk/lib/index.js:438:51799)

Building Assets Failed: Error: LambdaStack: SSM parameter /cdk-bootstrap/hnb659fds/version not found. Has the environment been bootstrapped? Please run 'cdk bootstrap' (see https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html)

Okay, you need to run npx cdklocal bootstrap first and then npx cdklocal deploy. The CLI shows you exactly the same as the CDK-CLI which is pretty cool. After deployment, you see in the Terminal something like

LambdaStack: creating CloudFormation changeset...

 ✅  LambdaStack

✨  Deployment time: 25.17s

Outputs:
LambdaStack.DemoRestApiEndpointF0DDDDE8 = https://i6d181wlsa.execute-api.localhost.localstack.cloud:4566/dev/
Stack ARN:
arn:aws:cloudformation:eu-central-1:000000000000:stack/LambdaStack/90063dd4

✨  Total time: 27.94s

If you have implemented an API Gateway like me, you will even see an endpoint as an Outputs. cdklocalcan even execute npx cdklocal watch and do hot-swap.

However, you may notice that it involved too many steps, and it's faster to use hot-swap directly with CDK.

AWS SAM local

The all-rounder AWS SAM can also run an emulator for its SAM template. It offers sam local with many subcommands:

• sam local generate-event – Generate AWS service events for local testing.

• sam local invoke – Initiate a one-time invocation of an AWS Lambda function locally.

• sam local start-api – Run your Lambda functions using a local HTTP server.

• sam local start-lambda – Run your Lambda functions using a local HTTP server for use with the AWS CLI or SDKs.

Strategy: Test-Driven Development (TDD)

Test-Driven Development (TDD) is an effective approach for breaking down Lambda code into smaller, single functions. This helps to structure the code better and gives the developer more confidence in the implementation. By breaking down your code into smaller, individual functions, you can easily write unit tests. Your Lambda handler should be the culmination of all your functions.

When using TDD, you can use a test runner such as Jest or Vitest for NodeJS, or pytest with pytest-watch for Python. This will help to ensure that the code is being tested for correctness and that any errors or bugs are quickly identified and addressed. Additionally, the test runner can be used to automate the testing process, saving time and effort.

When using TDD and Lambda, mocking AWS services can be challenging. Your Lambda's behavior may not match expectations, making it difficult to identify all edge cases. Despite this, the advantages of TDD are worth it, as it ensures the code works correctly and detects and resolves errors or bugs before they become a problem in production. This increases confidence after deployment.

Conclusion

We have seen some strategies for developing Lambda faster by leveraging tools that are bypassing Cloudformation such as SAM Accelerate or CDK's hotswap. It is also okay to gain confidence by using local emulators as they can be integrated into your development workflow.

Let's summarize

According to Yan Cui's blog post titled 'My testing strategy for serverless applications', using an emulator like localstack is a waste of time and breaks in a mysterious way. However, using a combination of different testing strategies can increase confidence before deployment. I have personally found that using TDD and hot-swap is effective for faster integration testing.

Let me know, what is your development setup and what tips and tricks you have.

n8n (pronounced: nodemation) is a low-code automation platform that empowers users to build powerful workflows and automate their daily tasks. It is an open-source tool and a very good alternative to Zapier or Make. However, these tools require you to know a bit of coding.

You can use it to automate (boring) tasks, you can save yourself time and energy while keeping your inbox organized and under control. In this blog post, I will show how you can declutter your inbox by labeling your mails by the respective sender and achieving a Zero Inbox.

Problem

The problem is simple, you might have a Gmail account since forever, and you have old emails which you probably will never read again. But they are in your inbox.

Another problem is that you might have too many unread E-Mails, and it would be nice to identify important emails and send the unimportant ones directly to the trash. By labeling them, you can then decide for example to unsubscribe from that newsletter.

What also can happen, is that you might reach the end of your storage. At that time, you will be forced to delete older emails. That leads to a struggle by searching which one to keep and which not... sigh... wish you would have categorized or tagged, right? Or maybe you don't trust Gmail and want to keep as fewer data as possible.

It is like your daily washing routine, keep good hygiene. In the following sections, we will set up an n8n workflow that will read messages, get their senders, use that value to create a label, and attach these to the messages.

Pre-requisite

If you follow along, you need to have n8n installed (I use Docker, but there is also a Desktop version) and a Google Developer account (use the tutorial of n8n to create your credentials).

You should also get familiar with n8n and also have some JavaScript knowledge. Luckily, n8n provides courses to get started.

Workflow

Below you can find my result workflow.

In order to use the Gmail- node, you need to create Gmail Credentials (Below is Google Calendar but it's pretty much the same).

The next subsections will explain each node.

Trigger

This is how every n8n- workflow starts. You can trigger a workflow on a scheduled basis, via webhooks, or in my case manually.

Pro-tip: n8n also offers a trigger whenever an E-Mail has been received. That would label your email immediately 😉

Get Messages

Once you are authenticated, you should be able to use all the functions from the Gmail Node. I recommend using a Limit and execute in batches. I also put a filter but you could leave that out and receive the latest 500 mails. Here it labels the messages which are in your 'INBOX'. Later you should archive your mails otherwise they would be fetched again.

Get the sender/domain name of the 'From'- value

This node contains a code snippet that extracts the sender domain from the From- value (E-Mail). I save that information into a new column and to avoid a naming clash later, I also renamed the id to mailId and removed the id - column.

for (const item of $input.all()) {
  // This is how 'From' looks like: Foo Bar <foo@bar.xyz>
  // and this method returns bar.xyz
  const sender = item.json.From.split('@')[1].slice(0,-1);
  item.json.sender = sender;
  item.json.mailId = item.json.id;
  delete item.json.id;
}

return $input.all();

Creating Labels

From the previous node, the value sender here will be set as a new Label. Important: the node should continue running regardless if the label already exists. If you don't enable the settings, the node might fail. For setting the label's name, we use the Expression function of n8n. That allows us to use a dynamic value depending on the sender's name.

Get all the labels

The next node is for retrieving all the labels.

But make sure that this is executed just once! n8n is implemented in a way that it runs for each row the Get All Labels- node. In my case, it would execute that node 500x.

Merge

The Merge- Node combines the mails with the labels accordingly. So the upper path creates the labels according to the sender's information. Each label then gets a new LabelID which needs to be combined with the E-mail. Below, you can find my option to achieve that.

Attach the Labels to the Mail

Now, we attach the label with the message by its Message ID and Label Id. Again here, we use the n8n's expression function. Remember that [above](# Get the sender/domain name of the 'From'- value), we have renamed the ID of the message to mailIdand the id below is the ID of the label.

Review each label and then delete the corresponding labels

After each message got attached, it's now human intervention to identify which E-Mails you should safely delete and which you should archive. Below, you can see a preview of the attached labels and in the next section, I will list which kind of E-Mails you can safely delete.

Ideas of which kind of E-Mails to delete

Great, now you labeled many emails, and it's now up to you what kind of emails to delete. I think it's good to label them first so that you as a human can now decide which emails to delete. Are you a bit overwhelmed, what to delete? Here are some emails you can delete for sure:

• Confirmation mails as they are outdated

• Notification mails by Google, Post courier, or any other providers which say e.g. that you have logged into a new browser as they just notify you

• RSVP of an invitation, well they are kinda Notification mails

• Messages which you also can find on their website or app (e.g. PayPal)

• Newsletter, by that time they are outdated

• Travel ternaries, if you haven't gone yet, why should you keep them? Hope you have taken great pictures

• Auto replies

Things to keep in mind

These are things you should keep in mind when building such a workflow.

You cannot run this workflow on all your E-Mails at once

Google might warn because of suspicious activities. They might also lock your account.

Recommendation: Run your workflow on 500 mails and do it twice a day. Also put a time filter here

Evaluate each label and try not to delete important mails

It is important not blindly to delete the emails but also to do a good review and not just a quick scan through them. Otherwise, you might delete important emails.

Put the mails into the archive

If you don't put the emails in the archive after reviewing them, they might be picked up again, and you will never finish decluttering your inbox.

The easiest way is to put everything to archive. However, that would still lead to some mails to archive which you might will never need again or which are simply outdated.

Full Workflow as Code

Below, you find the whole workflow as JSON which you can paste into your n8n.

{
  'meta': {
    'instanceId': 'e842ae497bd2320305776c95ab48327e7541feb0365d1ecd05e0e0ca871ef6df'
  },
  'nodes': [
    {
      'parameters': {},
      'id': '94603d9d-1f1a-4d77-8d0a-f1cfa9c0a7ab',
      'name': 'When clicking \'Execute Workflow\'',
      'type': 'n8n-nodes-base.manualTrigger',
      'typeVersion': 1,
      'position': [
        940,
        620
      ]
    },
    {
      'parameters': {
        'mode': 'combine',
        'mergeByFields': {
          'values': [
            {
              'field1': 'name',
              'field2': 'sender'
            }
          ]
        },
        'options': {
          'clashHandling': {
            'values': {
              'resolveClash': 'preferInput1'
            }
          }
        }
      },
      'id': '81cb45ca-bccb-4c21-95fb-6db89969acfc',
      'name': 'Merge',
      'type': 'n8n-nodes-base.merge',
      'typeVersion': 2,
      'position': [
        1980,
        600
      ]
    },
    {
      'parameters': {
        'operation': 'getAll',
        'limit': 500,
        'filters': {
          'labelIds': [
            'INBOX'
          ],
          'receivedBefore': '2022-02-28T23:00:00.000Z'
        }
      },
      'id': '14b681ff-4ce8-4b5e-bc71-1c7cac34d13e',
      'name': 'Get Messages',
      'type': 'n8n-nodes-base.gmail',
      'typeVersion': 2,
      'position': [
        1160,
        620
      ],
      'notesInFlow': true,
      'credentials': {
        'gmailOAuth2': {
          'id': '8',
          'name': 'Gmail account'
        }
      },
      'notes': 'Messages which have the Label \'INBOX\''
    },
    {
      'parameters': {
        'jsCode': 'for (const item of $input.all()) {\n  const sender = item.json.From.split('@')[1].slice(0,-1);\n  item.json.sender = sender;\n  item.json.mailId = item.json.id;\n  delete item.json.id;\n}\n\nreturn $input.all();'
      },
      'id': '209f6aeb-ca56-4b27-8cf8-f5d3a029f523',
      'name': 'Get the Sender and Cleaning',
      'type': 'n8n-nodes-base.code',
      'typeVersion': 1,
      'position': [
        1380,
        620
      ],
      'notesInFlow': true,
      'notes': 'The function which slices the value from the \'From\'- value to get the sender/its domain'
    },
    {
      'parameters': {
        'resource': 'label',
        'returnAll': true
      },
      'id': 'daab0c5b-31bd-47c1-9054-fa40709c545b',
      'name': 'Get All Labels',
      'type': 'n8n-nodes-base.gmail',
      'typeVersion': 2,
      'position': [
        1760,
        400
      ],
      'executeOnce': true,
      'credentials': {
        'gmailOAuth2': {
          'id': '8',
          'name': 'Gmail account'
        }
      }
    },
    {
      'parameters': {
        'operation': 'addLabels',
        'messageId': '={{ $json.mailId }}',
        'labelIds': '={{ $json.id }}'
      },
      'id': '1d3c4cca-bb1f-46ee-9d13-d8fc9065b02e',
      'name': 'Attach Label to Mail',
      'type': 'n8n-nodes-base.gmail',
      'typeVersion': 2,
      'position': [
        2200,
        600
      ],
      'credentials': {
        'gmailOAuth2': {
          'id': '8',
          'name': 'Gmail account'
        }
      }
    },
    {
      'parameters': {
        'resource': 'label',
        'operation': 'create',
        'name': '={{ $json.sender }}',
        'options': {}
      },
      'id': '44a02846-31a5-42e3-af1e-7378ff4f87f0',
      'name': 'Creating Labels',
      'type': 'n8n-nodes-base.gmail',
      'typeVersion': 2,
      'position': [
        1560,
        400
      ],
      'alwaysOutputData': true,
      'retryOnFail': false,
      'credentials': {
        'gmailOAuth2': {
          'id': '8',
          'name': 'Gmail account'
        }
      },
      'continueOnFail': true
    }
  ],
  'connections': {
    'When clicking \'Execute Workflow\'': {
      'main': [
        [
          {
            'node': 'Get Messages',
            'type': 'main',
            'index': 0
          }
        ]
      ]
    },
    'Merge': {
      'main': [
        [
          {
            'node': 'Attach Label to Mail',
            'type': 'main',
            'index': 0
          }
        ]
      ]
    },
    'Get Messages': {
      'main': [
        [
          {
            'node': 'Get the Sender and Cleaning',
            'type': 'main',
            'index': 0
          }
        ]
      ]
    },
    'Get the Sender and Cleaning': {
      'main': [
        [
          {
            'node': 'Creating Labels',
            'type': 'main',
            'index': 0
          },
          {
            'node': 'Merge',
            'type': 'main',
            'index': 1
          }
        ]
      ]
    },
    'Get All Labels': {
      'main': [
        [
          {
            'node': 'Merge',
            'type': 'main',
            'index': 0
          }
        ]
      ]
    },
    'Creating Labels': {
      'main': [
        [
          {
            'node': 'Get All Labels',
            'type': 'main',
            'index': 0
          }
        ]
      ]
    }
  }
}

Conclusion

This n8n- workflow declutters your inbox by creating labels for the senders. The owner can then identify quicker which emails to be deleted and which one to keep/archive. This approach usually takes a lot of time, and building a workflow like this saves you a lot of time. But keep in mind that this workflow does not delete, but just labels them. However, it took me weeks to declutter my inbox, as you cannot label it all at once. Take the section 'Things to keep in mind' seriously.

The workflow is also adaptable to any other workflow automation tool, such as Zapier and Make.

It's like your body's hygiene. The cleaner and tidier your inbox is, the more relaxed and mind will be. Start your healthy digital hygiene now! If you want to integrate that yourself but struggle with the setup, I am happy to help. Follow for more!

CDK pipelines integrate with other AWS services, such as AWS CodePipeline and AWS CodeBuild, to enable you to use their functionality within your CDK pipelines. For example, you can use AWS CodePipeline to manage the release process for your application, and AWS CodeBuild to build and test your application.

Overall, CDK pipelines provide an easy way to automate the deployment of your CDK-based applications, helping you to save time and effort and ensure that your applications are deployed consistently and reliably.

In this blog, I will show you how to use CDK pipelines to create a Codepipeline for deploying CDK-Stack into multiple accounts.

Why Have Multiple AWS Accounts?

Using multiple AWS accounts can help you to effectively manage your resources, isolate your environments and business units, and control access to your resources. In addition, it can enhance your security and help you comply with regulatory requirements. It is recommended to use separate accounts for different purposes from the start or gradually migrate your resources from a single account to multiple accounts.

Here are some keys to using multiple accounts

• Improved resource management

• Increased security

• Compliance with regulations

The Setup

If you're setting up Landing Zones, consider using superwerker. This tool helps set up AWS Control Tower and AWS Organization for account management, following best practices. It's also best practice to have separate accounts for Dev, Staging, and Prod for your application, as well as an account for your CI/CD, which may contain your Git repository and AWS ECR for your Docker images.

Assuming you've set up separate accounts, you can develop a CDK stack and push the code to your CI/CD account. From there, any CI/CD pipeline should be triggered. To enable this, use AWS CodePipeline and the CDK library: CDK pipelines.

Below you will find the setup for this blog.

The CDK- pipeline Stack

It's important to note the difference between the Codepipeline module and CDK Pipelines. While CDK Pipelines uses CodePipeline for deploying stacks created with CDK, CodePipeline is a more general-purpose tool that allows you to customize your own pipeline as you would when setting up a new CI/CD pipeline.

With CodePipeline, you can define the entire release process for your application, including building, testing, and deploying your code. This flexibility allows you to tailor your pipeline to your specific needs, which can be especially useful if you have specific compliance or security requirements.

On the other hand, CDK Pipelines provides a more streamlined approach to deploying CDK-based applications. It automates the deployment process and integrates with other AWS services, such as CodeBuild, to help you build, test, and deploy your applications.

Overall, both CodePipeline and CDK Pipelines provide valuable tools for automating the deployment of your applications. Understanding the differences between them can help you choose the best tool for your specific use case. You can find more information about Codepipeline in the official AWS CDK documentation.

Defining a CDK- Pipeline

To define a CDK pipeline in our PipelineStack, we will use the CodePipeline construct from aws-cdk-lib/pipelines. While there are more properties available (which can be found in the documentation), this blog post will focus on deploying a simple CDK stack into multiple accounts. The only required property is synth.

The build step that produces the CDK Cloud Assembly.

I must admit that I'm not sure what that means. Plus, simply reading the name IFileSetProducer doesn't provide much context. However, based on the example provided, it appears that the ShellStep is leveraging the functionality of the IFileSetProducer interface. Another option could be to use a CodeBuildStep, since the pipeline is using CodeBuild for both installation and building. For the sake of this blog, we'll stick with the ShellStep.

The ShellStep requires an input of type CodePipelineSource, which represents the source of our CodePipeline. The source could be Github, CodeCommit, or S3. In this case, we will use an existing CodeCommit repository.

For the commands, there are default commands yarn install and yarn build. You can add testing or linting commands, or change the package manager as needed. The pipeline uses CodeBuild with all the necessary tools for deploying CDK stacks such as yarn.

One important property for cross-account deployment is crossAccountKeys. It is a Boolean and should be set to true, even if you are not doing cross-account deployment eventually you might plan to do cross-account deployment.

Create KMS keys for the artifact buckets, allowing cross-account deployments.

The stack has a property pipeline that you can then access later.

import { Stack, Fn, StackProps } from 'aws-cdk-lib';
import * as codecommit from 'aws-cdk-lib/aws-codecommit';
import { CodePipeline, CodePipelineSource, ShellStep } from 'aws-cdk-lib/pipelines';
import { Construct } from 'constructs';

export class PipelineStack extends Stack {
  readonly pipeline: CodePipeline;
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);
    // Existing CodeCommit
    const repository = codecommit.Repository.fromRepositoryName(this, 'CodeCommit', 'my-cool-codecommit');
    this.pipeline = new CodePipeline(this, 'CdkPipeline', {
      synth: new ShellStep(`${props.pipelineName}Synth`, {
        input: CodePipelineSource.codeCommit(repository, props.branch),
        commands: props.commands ?? [
          'yarn install',
          'yarn build',
        ],
      }),
            crossAccountKeys: true,
    });
  }
}

Adding a Stage for Deploying Stack

Now, we can use our PipelineStack into main.ts . Since we added a readonly pipeline , we can now add stages to the pipeline.

The method addStage() requires a class that extends from Stage . Here, we named it MyApplication and it only has StackProps. Now inside MyApplication, we can add a new Stack, MyStack where define your resources which you then deploy to a different account. The MyApplication requires an env- object to deploy to different stages by giving it an account number and a region.

// In main.ts

import { App, Stage, Stack } from '@aws-cdk/core';
import { PipelineStack } from './pipeline-stack';
import { ManualApprovalStep } from 'aws-cdk-lib/pipelines';

const app = new App();

const cicd = new PipelineStack(app, 'PipelineStack', {
  env: { account: 'MySharedAccount', region: 'us-east-2' },
});

class MyApplication extends Stage {
  constructor(scope, id, props) {
    super(scope, id, props);

    new MyStack(this, 'MyStack');
  }
}

export class MyStack extends Stack {
  constructor(scope, id) {
    super(scope, id);

        /* My resources */
  }
}

cicd.pipeline.addStage(new MyApplication(app, 'DevDeployment', {
  env: {
    account: 'DEV_ACCOUNT',
    region: 'eu-west-1',
  },
}));


cicd.pipeline.addStage(new MyApplication(app, 'TestDeployment', {
  env: {
    account: 'TEST_ACCOUNT',
    region: 'eu-west-1',
  },
}));

cicd.pipeline.addStage(new MyApplication(app, 'TestDeployment', {
  env: {
    account: 'PROD_ACCOUNT',
    region: 'eu-west-1',
  },
}, {
  pre: [
    new ManualApprovalStep('PromoteToProd'),
  ],
}));

app.synth();

Bootstrapping

Let's take a look at what we have achieved up to this point. We have multiple accounts set up and we will deploy the CDK from above in our CI/CD- account. This account will then deploy to our Dev, Test, and Prod - environment.

In order to do that, each Development environment needs to trust the CI/CD- account. The following script should be executed once to allow the CI/CD- account to deploy to each development account. Below you find a command on how to trust a different account. In this case, you trust the CICD_ACCOUNT from a DEV_ACCOUNT , TEST_ACCOUNT , and PROD_ACCOUNT. MY_SERVICE_ACCESS_POLICY can be for example AWSCodeDeployFullAccess , AWSEcsFullAccess or a custom policy. You can chain them accordingly with a comma (see below).

# Replace *_ACCOUNT accordingly to your account number
CICD_ACCOUNT="replace-this"
DEV_ACCOUNT="replace-this"
TEST_ACCOUNT="replace-this"
PROD_ACCOUNT="replace-this"
MY_SERVICE_ACCESS_POLICY1="replace-this"
MY_SERVICE_ACCESS_POLICY2="replace-this"
AWS_REGION="eu-west-1"

npx cdk bootstrap \
  --profile dev-account-profile \
  --trust $CICD_ACCOUNT \
    --cloudformation-execution-policies arn:aws:iam::aws:policy/$MY_SERVICE_ACCESS_POLICY1,arn:aws:iam::aws:policy/$MY_SERVICE_ACCESS_POLICY2 \
  aws://$DEV_ACCOUNT/$AWS_REGION

npx cdk bootstrap \
  --profile test-account-profile \
  --trust $CICD_ACCOUNT \
    --cloudformation-execution-policies arn:aws:iam::aws:policy/$MY_SERVICE_ACCESS_POLICY1,arn:aws:iam::aws:policy/$MY_SERVICE_ACCESS_POLICY2 \
  aws://$TEST_ACCOUNT/$AWS_REGION

npx cdk bootstrap \
  --profile prod-account-profile \
  --trust $CICD_ACCOUNT \
    --cloudformation-execution-policies arn:aws:iam::aws:policy/$MY_SERVICE_ACCESS_POLICY1,arn:aws:iam::aws:policy/$MY_SERVICE_ACCESS_POLICY2 \
  aws://$PROD_ACCOUNT/$AWS_REGION

Deploy Stack

Let's take a look at what we have achieved up to this point. We have multiple accounts set up and we will deploy the CDK from above in our CI/CD- account. This account will then deploy to our Dev, Test, and Prod - environment.

In order to do that, each Development environment needs to trust the CI/CD- account. The following script should be executed once to allow the CI/CD- account to deploy to each development account. Below you find a command on how to trust a different account. In this case, you trust the CICD_ACCOUNT from a DEV_ACCOUNT , TEST_ACCOUNT , and PROD_ACCOUNT. MY_SERVICE_ACCESS_POLICY can be for example AWSCodeDeployFullAccess , AWSEcsFullAccess or a custom policy. You can chain them accordingly with a comma (see below).

Conclusion

In this blog post, you have seen how to deploy to several accounts with CDK- pipelines in a multi-account setup. The CDK- pipelines provide an easy and efficient way to automate the deployment of CDK-based applications.

As always, the setup is always a bit tedious but once you've done all that, you have a good process when deploying CDK stacks to multiple accounts. This will save a lot of time and effort and can help your organization scale its cloud operations. Plus, thanks to CDK you can build a CDK library and share the construct among your developers.

This was part one of how you can achieve cross-account deployment using CDK pipelines. The next part is how you can leverage the power of Typescript to deploy a generic CDK stack.

Thanks for reading and happy coding!