TL;DR

• A terraform apply killed mid-run in GitHub Actions leaves behind two DynamoDB artefacts: a stale lock and a mismatched MD5 digest.

• Most guides only mention force-unlock. That fixes the lock, but you'll still get "state data in S3 does not have the expected content" until you patch the digest.

• This post walks through the why, the diagnosis, and the exact 7-step fix so you can recover cleanly without recreating state from scratch.

The Incident

I was rolling out ECR repositories for four microservices via a reusable Terraform module. The pipeline, a standard plan → apply workflow on GitHub Actions had been reliable for months.

One afternoon the CI runner was terminated mid-apply. The reason didn't matter much (runner preemption, timeout, OOM — pick your favourite). What mattered was the aftermath: every subsequent terraform plan failed with this:

Initializing modules...
- orders_api_service_ecr_repo      in ../../../modules/aws_ecr
- notifications_service_ecr_repo   in ../../../modules/aws_ecr
- inventory_service_ecr_repo       in ../../../modules/aws_ecr
- gateway_service_ecr_repo         in ../../../modules/aws_ecr

Initializing the backend...

Successfully configured the backend "s3"!

Error refreshing state: state data in S3 does not have the expected content.

This may be caused by unusually long delays in S3 processing a previous state
update. Please wait for a minute or two and try again. If this problem
persists, and neither S3 nor DynamoDB are experiencing an outage, you may need
to manually verify the remote state and update the Digest value stored in the
DynamoDB table to the following value: a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6

Terraform told me what to do, update a Digest, but not where or why. If you’ve landed here from the same error, read on.

How the S3 Backend Actually Works

Before jumping to the fix, it helps to understand the moving parts. Terraform’s S3 backend uses two AWS services in tandem:

Key insight — DynamoDB stores two items per state file, not one:

When apply finishes normally, Terraform:

1. Writes the new state to S3.

2. Computes the MD5 of that file and stores it in the -md5 item.

3. Releases the lock by deleting the lock item.

When the runner is killed mid-apply, steps 2 and 3 never happen. That leaves you with two problems, not one.

Diagnosis: Two Problems, Not One

Problem 1: Stale Lock

The lock item at …/terraform.tfstate was never released because the runner was killed. Any future plan or apply will fail with "state is locked".

Problem 2: Digest Mismatch

The interrupted apply may have written a partial or updated state file to S3, but the MD5 in the -md5 DynamoDB item still reflects the previous state. Terraform computes the MD5 of the current S3 object, compares it to the stored digest, and refuses to proceed because they don't match.

Most Stack Overflow answers jump straight to force-unlock. That fixes Problem 1 but leaves Problem 2 untouched, and you can't even run force-unlock until init succeeds, which it won't until the digest is fixed.

The 7-Step Recovery

Step 1: Confirm nothing is running

Check GitHub Actions for any in-flight runs of your apply workflow. Check local terminals too. Running force-unlock while a legitimate operation is in progress will corrupt state.

Step 2: Back up the S3 state file

In the S3 bucket, locate global/ecr/terraform.tfstate (or your equivalent key):

• Verify it exists and is non-zero.

• If S3 versioning is enabled, download the current and previous version. The current one may be partially written.

aws s3 cp s3://your-bucket/global/ecr/terraform.tfstate ./terraform.tfstate.bak

Step 3: Patch the digest in DynamoDB

Open DynamoDB → your lock table → Explore items. Search for the item whose LockID ends with -md5:

your-bucket/global/ecr/terraform.tfstate-md5

• If the item exists: update its Digest attribute to the value from the error message (e.g. a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6).

• If it doesn’t exist: create a new item with LockID = …-md5 and Digest = that hash.

Why this value? Terraform already computed the MD5 of the current S3 object and told you in the error. You’re simply telling DynamoDB “yes, that’s the right file.”

Step 4: Run terraform init

terraform init

This should now succeed. If it still fails with the digest error, double-check the LockID key — the path must exactly match.

Step 5: Force-unlock the stale lock

terraform force-unlock <LOCK-ID>

The lock ID is the UUID from the lock item’s Info JSON. Terraform will prompt for confirmation.

Step 6: Plan and review

terraform plan

Review carefully. Some resources may have been created by the interrupted apply. The plan shows exactly what’s pending.

Step 7: Apply

terraform apply

Why Order Matters

You cannot skip ahead. init needs a valid digest. force-unlock needs a successful init. plan/apply need the lock released. The dependency chain is strict.

Preventing This Next Time

A few guardrails I’ve added since this incident:

1. S3 versioning: Always enabled on the state bucket. Gives you a rollback path if the state file itself is corrupted.

2. CI timeouts with grace periods: Set workflow timeout-minutes generously and add a cleanup step that logs the lock ID on failure.

3. Alerting on stale locks: A simple scheduled Lambda that scans the DynamoDB lock table for items older than N hours and posts to Slack.

4. State backup before apply: Add a pre-apply step in CI that copies the current state to a versioned “backup” prefix in S3.

Note on Terraform 1.10+: Terraform now supports S3-native state locking without DynamoDB. If you’re starting fresh, consider this path, the digest/lock split issue goes away entirely.

References

• Terraform S3 Backend Documentation: Official backend config reference including new S3-native locking.

• terraform force-unlock Command: CLI reference for manual lock removal.

• GitHub Issue #20708: Community thread on the exact “state data does not have expected content” error.

• Terraform State Corruption Recovery (Medium): A complementary deep dive on state corruption scenarios.

• Managing Terraform State on AWS (Terrateam): Solid end-to-end guide on S3 + DynamoDB setup with GitHub Actions.

Thank you for reading this article! 🙏 If you’re interested in DevOps, Security, or Leadership for your startup, feel free to reach out at hi@iamkaustav.com or book a slot in my calendar.

👉 Don’t forget to subscribe to my newsletter for more insights on my security and product development journey. Stay tuned for more posts!

💡 One shameless promotion: I’m building an easy-to-use freelance management service for technical freelancers. Check it out here → https://www.getprismo.app/. If you are interested to secure limited seats of early adopters, Join the waitlist.

git config --global alias.switch-master 'checkout master'

With this alias in place, you can simply type git switch-master to switch to the master branch.

Also you can create symlink for these git commands, for an example -

git symbolic-ref refs/heads/master refs/heads/main

This method ensures that you can seamlessly navigate between the main and master branches, maintaining your workflow without any disruptions.

This tool won’t replace everything, but it can help you use old commands like:

git checkout master

git rebase -i master

git diff master

Thank you for reading this article! If you're interested in DevOps, Security, or Leadership for your startup, feel free to reach out at hi@iamkaustav.com or book a slot in my calendar. Don't forget to subscribe to my newsletter for more insights on my security and product development journey. Stay tuned for more posts!

Let’s assume the CTO of PlatformSecurityTech (an imaginary IT company) asked their Platform team to create a few accounts for their product offerings. Right after the request, the team set up 3 accounts per the above diagram and handed them to the company. The team used AWS Identity Center to manage its users securely and maintain further security.

Since users of Account A and Account B had unrestrictive permissions for their day-to-day needs (it could be a lack of understanding of programmatic access through SSO), many IAM users were created. Among them, a few were not even used for months 🤔. It was getting difficult to maintain their usage. The team decided to perform bi-weekly audits on existing IAM users in different accounts and later monitor them.

It’s easy if you have one or two accounts to monitor. But in an ideal world, there are many of them. Going through each one is not only tiring but also time-consuming. They decided to go ahead with Advance Query in AWS Config to increase productivity and provide a seamless experience in monitoring those IAMs.

Given they restructured the organization as below,

Here, they developed another AWS account called Delegated Admin, which manages all the administrative work (non-root account). Due to its nature, this new account is connected with all other accounts and can fetch data and perform specific activities in those accounts. Assuming AWS Config is configured in the new account, you can now observe your account’s activity. Now, to check the existing IAM users,

Go to https://console.aws.amazon.com/config/ for the AWS Config dashboard. There, you will find a similar-looking sidebar. Now, go to Advance Query from the sidebar.

You will find something similar to the above page, which might be pre-populated with a few queries through the team-configured conformance pack. Click on the new query and add it to the query editor below.

SELECT
  accountId,
  resourceName,
  resourceId,
  resourceCreationTime
WHERE
  resourceType = 'AWS::IAM::User'
ORDER BY
  accountId

Running this query will show the list of IAM users in the AWS organization, including the other details mentioned in the selected query. Once the team collects the data from their audits, they can contact the creator of those IAM users for their needs and perform the corrections accordingly. One can export the data to any sharable format for frequent updates.

Thank you for reading this article! If you're interested in DevOps, Security, or Leadership for your startup, feel free to reach out at hi@iamkaustav.com or book a slot in my calendar. Don't forget to subscribe to my newsletter for more insights on my security and product development journey. Stay tuned for more posts!

As part of my exploration of Golang, I came across a popular feature: first-class support for concurrency. I believe we all understand the benefit or importance of concurrency. In the HTTP way, when an endpoint needs to fetch data from multiple upstreams, aggregate the data and produce it as a response, Go concurrency helps to reduce the latency for that API request. Two features in Go, goroutines and channels make concurrency easier when used together.

Goroutines example: Run functions in parallel

Modern computers are equipped with processors, or CPUs, designed to efficiently handle multiple streams of code simultaneously. These processors are built with one or more "cores," each capable of running one code stream at a given time. To fully utilize the speed boost multiple cores offer, programs must be able to split into various streams of code. This division can be challenging, but Go was explicitly developed to simplify this process.

Go achieves this through a feature known as goroutines, special functions that can run alongside other goroutines. When a program is built to execute multiple streams of code simultaneously, it operates concurrently. Unlike traditional foreground operations, in which a function runs to completion before the following code executes, goroutines allow for background processing, enabling the following code to run while the goroutine is still active. This background operation ensures that the code doesn't block other processes from running.

Goroutines provide the advantage of running on separate processor cores simultaneously. For instance, if a computer has four processor cores and a program has four goroutines, all four can run concurrently. This simultaneous execution of multiple code streams on different cores is called parallel processing.

Jumping into the example, create a multifunc directory named go-concurrency-project.

mkdir go-concurrency-project
cd go-concurrency-project

Once you’re in the go-concurrency-project Directory, open a file named main.go using nano, or the editor of your choice:

nano main.go

Add the following code to the main.go file,

package main

import (
    "fmt"
)

func make(total int) {
    number := 0
    for number < total {
        number = number + 1
        fmt.Printf("Generated number %d\n", number)
    }
}

func print() {
    number := 0
    for number < 2 {
        number = number + 1
        fmt.Printf("Print: number %d\n", number)
    }
}

func main() {
    print()
    make(2)
}

Based on the above setup, make and print Functions are structured to run in sequence. make Accepts a number to generate up to and prints only five numbers.

This is how it will look like when we execute main.go,

go run make.go

// Output
Print: number 1
Print: number 2
Generated number 1
Generated number 2

If you notice, the function printed the output in sequence based on its execution pattern.

When running two functions synchronously, the program takes the total time for both functions to run. But if the functions are independent, you can speed up the program by running them concurrently using goroutines, potentially cutting the time in half. To run a function as a goroutine, use the go keyword before the function call. However, you need to add a way for the program to wait until both goroutines have finished running to ensure they all complete running.

To synchronize functions and wait for them to finish in Go, you can use a WaitGroup from the sync package. The WaitGroup primitive counts how many things it needs to wait for using the Add, Done, and Wait functions. The Add function increases the count, Done decreases the count, and Wait can be used to wait until the count reaches zero.

To do that update main.go,

package main

import (
    "fmt"
    "sync"
)

func make(total int, wg *sync.WaitGroup) {
    defer wg.Done()

    number := 0
    for number < total {
        number = number + 1
        fmt.Printf("Generated number %d", number)
    }
}

func print(wg *sync.WaitGroup) {
    defer wg.Done()

    number := 0
    for number < 2 {
        number = number + 1
        fmt.Printf("Print: number %d", number)
    }
}

func main() {
    var wg sync.WaitGroup

    wg.Add(2)
    go print(&wg)
    go make(2, &wg)

    fmt.Println("Awaiting....")
    wg.Wait()
    fmt.Println("Done!")
}

After declaring WaitGroup, specify how many processes to wait for. In the example, the goroutine waits for two Done calls before finishing. If not set before starting the goroutines, things might happen out of order, or the code may panic because wg doesn't know if it should wait for any Done calls.

Each function will use defer to call Done, which decreases the count by one after the function finishes. The main function is updated to include a call to Wait on the WaitGroup. This ensures that the main function waits until both functions call Done before continuing and exiting the program.

After saving your main.go execute the file,

go run main.go

// Output
Awaiting....
Generated number 1
Generated number 2
Print: number 1
Print: number 2
Done!

Your output may vary each time you run the program. With both functions running concurrently, the output depends on how much time Go and your operating system allocates to each function. Sometimes, each function runs entirely, and you'll see their complete sequences. Other times, the text will be interspersed.

Conclusion

If you’re interested in learning more about concurrency in Go, the Effective Go document created by the Go team provides much more detail. The Concurrency is not parallelism Go blog post is also an exciting follow-up about the relationship between concurrency and parallelism. These two terms are sometimes mistakenly thought to mean the same thing.

Thank you for reading this article! If you're interested in DevOps, Security, or Leadership for your startup, feel free to reach out at hi@iamkaustav.com or book a slot in my calendar. Don't forget to subscribe to my newsletter for more insights on my security and product development journey. Stay tuned for more posts!

But fear not, my friend! In this blog post series, we will explore the exciting and essential world of DevOps security and how it's the key to safeguarding the success of early-stage startups. We'll take a deep dive into the unique challenges these startups face, the potential consequences of neglecting security, and the actionable steps they can take to mitigate risks and thrive in an ever-evolving market. So, let's grab a cup of coffee, buckle up, and join me on this thrilling journey where innovation meets safeguarding and where startups can unlock their full potential while staying one step ahead of the game. Let's go!

What is DevOps Security?

DevOps security, often called DevSecOps, is a crucial aspect of any early-stage startup's operations. With the integration of development and operations teams (DevOps), security becomes an inherent part of the entire cycle rather than an afterthought. This approach allows startups to innovate rapidly while minimising the potential risks associated with cybersecurity threats. This balance is crucial in today's digital age, where data breaches and cyber threats can cause significant damage to a company's reputation and resources. In the following sections, we will delve deeper into the importance of DevOps security in the landscape of early-stage startups.

Principles to secure product environments.

Collaboration, automation, and continuous monitoring enhance an organisation's security posture.

Collaboration

Collaboration integrates security considerations into every stage of software development, reducing overlooked vulnerabilities. Infrastructure experts and products need a solid partnership to handle all critical and significant security issues, i.e., password strength for authentication, access control on a cloud provider, security key accidentally pushed on VCS, etc.

Automation

Automation speeds up processes, minimises errors, and enables early detection of vulnerabilities by incorporating security checks and tests into the development pipeline. One of the best and minimalist examples is Dependabot issues on GitHub. The first step to take this leap would be fixing them as soon as they arise.

Continuous monitoring

Continuous monitoring allows for ongoing assessment of vulnerabilities and immediate response to issues, minimising opportunities for attackers.

These DevSecOps principles strengthen defences, mitigate risks, and contribute to a startup's success.

Story Time: Challenge faced by an early-stage startup

A few years back, I was collaborating with a startup in the Healthcare space. They were not early in the business but were a young engineering department. This company (Let's call this Meditech INV*- It is an artificial name and not related to any past or existing company*) built an extremely modern analytics and reporting SaaS platform. This platform caters to hospitals in a government program that provides healthcare benefits to a selected patient group. The market already has veteran players with relatively old-age tech. Meditech INV wanted to establish itself as a cutting-edge SaaS provider in its TAM.

On a fine morning, a news burst that one of Meditech INV's competitors & market leaders experienced a data breach. In healthcare, different countries have different lawsuits for such data breaches. This destroyed the competitor's reputation, and in the next few days, they started losing existing clients. This was an opportunity for Meditech INV to acquire those clients almost free (without any CAC). However, this incident changed the market's mood as clients now worry about their patient's data. They needed these SaaS platforms to demonstrate their security practices, and based on that, they would move forward with the contract. Even after 6 months of being in the market and working with 5-6 big hospitals, the product & infrastructure didn't have enough security observability. That means even if we had any data breach, we would barely know what had happened (however, we had pretty strong network rules in place since the beginning). This means we may lose all of these businesses that were organically coming our way.

It was a great weekend challenge. My phone rang. After 30 min, I was in front of my laptop to hack around security. We were setting up our first security modules after 6 months of going live. I remember our only principal engineer & I spent the precious Saturday setting up security tools on our AWS infrastructure (having 6 AWS accounts back then) and polishing them up Sunday. We were trying to rush for the new opportunities, with a baggage of fear that we may lose our existing clients if we couldn't show them some visualisation, which may result in us being out of business and closing the offices with ~30+ engineers and 400+ other roles working there. Scary right??????

The Impact

This situation created a panic among the senior leadership. Full of excitement but stressful days, I think that's what happens when you delay your security practices. We have taken these three primary measures,

• Audit every activity across AWS as a preventive measure.

• We are introducing compliance checks across all AWS accounts.

• We are guarding AWS so that we know when a malicious activity has happened.

• An informative dashboard with a security score that we can present to our existing and new clients to gain their trust.

Over a few weeks, the veteran company went out of business, and here we are with $400M+ revenue and 30+ new clients. We are looking forward to acquiring a renowned company. This incident teaches us a few exciting things,

• Security practices are vital from the beginning, not at the end of your feature development.

• Allocate some budget in every sprint to improve current security conditions. You don't know when opportunity will knock on your door.

• When a tech organisation grows with more than 2 products at scale, you need dedicated security experts. It can be in the same Platform team or a separate team.

• Most importantly, in a startup, and when you are in some answerable position, don't keep any advance plans for weekends :P.

The Conclusion

Early-stage startups must prioritise security from the get-go. As they navigate the complexities of DevSecOps, seeking guidance from experienced professionals can be immensely beneficial. By doing so, startups will foster a security-first culture that will protect their venture and boost its credibility and trustworthiness in the eyes of stakeholders and customers. So, never underestimate the power of security.

I help startups with DevOps, Security & Leadership. If you are interested / have requirement in any of them, or starting a startup and want a quick conversation, let's book time on my calendar or don't hesitate to contact me at hi@iamkaustav.com.

There will be few more post about my journey with security & product development. Please subscribe to my newsletter. Thanks for sticking with me so far :)

In my experience, starting working on Golang after almost a decade with NodeJS and JVM-based languages, was a significant step forward in my coding journey. Coming from a DevOps and automation background, Golang adds a significant focus on how you define the architecture of your project.

The speed

The speed of Golang can be attributed to several factors:

1. The Compilation Factor: Unlike VM-dependent languages, Golang directly compiles to the binary files. The direct compilation results in faster application development.

2. Automated Garbage Collection: Golang automatically handles garbage collection, which increases the speed significantly.

3. Concurrency First: This beautiful language is designed keeping high concurrency in mind. It's an ideal language for implementing data-heavy operations. The lightweight goroutine and channels communicate effectively between program segments, resulting in a highly performant application.

So far in my career, running tests and building took ~5-7 mins for fat products, bringing that down to 3 min was a big achievement and we were okay with it in JVM-based languages. But it took my breadth away to see unit/integration tests took a few seconds to complete those executions.

These factors combined make Golang a highly optimized language, capable of executing tasks quickly and efficiently.

Learning and Development

1. Fewer Set of Features: Golang has a very minimal set of core features to get the job done, which reduces the learning curve and helps you to onboard as soon as possible.

2. The Clean Code Approach: Often depending on how you write, writing Golang code is clean and easy to understand. It simplifies debugging and improvement of code written by others.

3. Inline Documentations: It can be easily documented using docstrings that are simple for beginners to remember.

Due to the above key reasons and plenty of others around learning and development, Golang is now one of the popular choices for building core automation and heavy backend work.

Effective Compiled Language

1. Easy to Compile: On compilation of the Golang program, it becomes a self-sufficient executable binary. This executable can be run on the targeted system without anything additional.

2. Performance: Being a compiled language, Golang is much faster than interpreted languages.

3. Statically Typed: Golang is a statically typed language. It checks each variable at compile-time, which can lead to more efficient code and fewer runtime errors.

What happened when I picked it?

I was super critical and concerned when I picked Golang. Pointers, struct etc and working with super experienced folks working with Golang for 5+ years or more, it's quite challenging to bring myself up to speed. Taking small steps and emptying my past knowledge and restrictions helped me to fall in love with this beautiful language and it's ecosystem.

TDD, clean code, and refactoring while understanding the key features like goroutines, and channels add icing on the cake and that's what makes Golang stand out from other languages.

Conclusion

I was an extreme NodeJS fan when it came to quick development and JVM-based languages for heavy data processing API services until I met Golang. This path crossed during my experiments with terraform providers. However, when I started writing code on Go for Rest APIs, it gave me a different experience and now quite difficult to move back to other languages unless there is a heavy need.

It is a powerful revolutionizing language for backend development. Its speed, simplicity, and scalability make it an excellent choice for developers looking to build efficient and robust applications. Concurrency, garbage collection, network API, and speed, Golang provides a comprehensive toolset for backend development. As we continue to see the rise of Golang in the tech industry, one thing is clear, Golang is an experience that every backend developer wants to have.

Thanks for reading the article. If you are interested in building something exciting together, let's have a chat. You can also find the services that I provide here.

The Story

Before we jump in into the list, let me tell you a story from my experience -

Seeing how startups build their product with limited developers is always interesting. I was part of a startup that didn't have a QA (Quality Analyst) initially. As a company, we were trying to move from a brick-and-mortar business to a digital company with our very first SaaS product. It's always a risk because it will be an initial setback if the first product fails. The developer group ensured we tested each feature properly before going live. A few days before THE SPECIAL DAY, we drafted a detailed checklist to ensure we considered all possible items to support our go-live, and if all of them are marked positive, then only we will make the production deployment. This process helped us successfully deploy the application, mark the product as a massive success for the organization, and support the product as a monolith with almost 30+ developers and 3+ teams working on it for over 2 years. We used to call them production checklists.

Production Readiness Checklist

From OpsLevel,

Production readiness checklists can reduce the cognitive load of having to remember all the different vulnerability and failure points we need to consider in our complex landscape.

To simplify it, the Production Readiness Checklist is the universal language for your team to answer, "Is the product ready for launch?" However, the checklist may vary between domains or teams. It may be possible to have different checklists for different tiers of an application. It gives a positive start to look into pending items before GoLive.

Production Readiness Checklists can be opinionated. Below are my recommendations based on the nature of the products I have dealt with.

General

1. Ownership: Ownership of the service is clearly defined. This can be achieved by having the CODEOWNERS file on your Git project or assigning contributors to a repository.

2. Documentation: It can cover multiple items, such as API documentation, development steps, README, onboarding instruction to the service and FAQs. Once the service is mature enough and if any significant defect is found, it can be documented in a troubleshooting guide, preferably staying on the repository.

3. SLA / SLO: Since this product was a customer-facing SaaS product, it is highly important to provide a clear SLA (Service Level Agreement) to the customer and SLO (Service Level Objectives) for the internal teams. Depending upon the business violation of SLA impacts the revenue of the company, so it is important to provide a realistic than optimistic SLA.

Data Management

1. Persistency: Ensure to have resiliency setup for the failure of a data source without loss of performance. It must have a continuous backup and a one-click restore mechanism.

2. Scaling: Strategy to scale the data source during optimum load.

Testing

1. Unit Tests: Unit tests are highly important for any service to operate with an additional safety net.

2. Integration Tests: When one is working with multiple components inside a single repository, for example, we had a multi-module structure connecting Controller, Services and Repository layer, it is important to have integration tests covering the contracts.

3. E2E Acceptance Test: End-to-end (E2E) tests are costly tests but extremely helpful when you are low in the availability of talent. When we were supposed to go live with our first release, we drafted a basic E2E test covering all of the business critical components.

4. Operational Testing: There are a few optional business or customer-level tests that can be added as an extra safety check for your application to avoid any unpleasure circumstances. Tests for worse customer experience, load tests, and sanity checks help to make your product operation ready.

It is recommended to execute these tests periodically on your CI/CD environments.

It is important to structure your automated tests well to achieve larger benefits. Here are few articles that helps with few ideas based on industry best practices,

• Guidelines for Structuring Automated Tests

• Integration test by Martin Fowler

Deployment

1. Continuous integration: Each repositories must have their automated pipeline, When an engineer pushes their changes, it runs the tests, builds, static code analysis, security checks etc. Based on the scale of the application, the team can decide what kind of automated steps they want to have in their pipeline.

2. Continuous Delivery: Depending upon organization standards or repository needs, one may deploy from lower-level environments to higher environments. As a standard, CD should execute after CI tasks (i.e. test, lint and build). Single-click deployments are quite popular and provide ease of realising new changes and downgrading applications. Changelog and Release Notes are mandatory after each production deployment.

Operational Excellence

1. Escalation Strategy: Escalation strategy itself is a big topic, but to keep it short, there are a few things one needs to set for each product/service,

   1. On-Call Rotation

   2. Incident Management with practices for short-term and long-term remediation

   3. Post Mortem of incidents

2. Runbooks: Runbooks are essential to understanding known failures and their quick remediations. Ensure it is up-to-date.

3. Observability: This contains multiple touchpoints,

   1. Logging

   2. Tracing

   3. Metrics

   4. Customer Impact / Operational Dashboard

   5. SLO/SLA dashboard

   6. Error Budget dashboard

Security

1. Authentication/authorization: Ensure authentication and authorization strategies are taken place for any public-facing API, regardless of internal or external apps.

2. Secrets: It is recommended to use a secret management store to keep all secret information such as database passwords. Recommendations are Vault, AWS Secret Manager etc.

3. Dependency Scan: To ensure all dependencies are up-to-date with their security fix. My goto tool is dependabot, however, I used Synk for dependency scan and other security practices, and have had positive experiences.

There are many other items that can be included based on the practices a team/organization is following. I would recommend structuring the checklist based on product and grouping them in a one-time or repetitive list. As an organization when a system matures, we must update the checklist. In our case, we kept updating our touch points every time we learned something new and focused on the automation-first approach. This helped us to ship our product with low defects that earned the trust of our users.

I prefer to keep the checklist as close as possible to the source of the product, usually at the GitHub repository in markdown format. This helps us to build automation if needed.