CartonCloud + Localstack + Docker. Our experience

CartonCloud, like many large, long-running software projects contains a mixture of technologies. The application was initially built as a standalone PHP application, however since 2018, various pieces of functionality have been built or ported into microservices and serverless applications.

As we’ve progressed through this journey, our staging, QA and production infrastructures became more complex, shifting from a PHP app that did it all, to multiple software languages and a large number of AWS services (SNS, SQS, S3, Kinesis, Lambda, Dynamo etc).

A growing issue within our development team had been our inability to easily replicate production behaviour when developing locally. Locally testing things like the interplay between two services via SQS/SNS could only be done by using our QA environments, and debugging remotely was particularly time consuming and complex to achieve. In addition, setting up a new development environment from scratch had become a painful, multi-day exercise for new starters. The how-to guide was so long that many went into shock on their first day as they tried to follow the hundreds of step-by-step instructions without making any mistakes.

While very difficult to get fixed figures on this, I estimate that prior to Localstack, approximately 10-15% of our teams total working time was being consumed trying to workaround local environment limitations, or debugging problems with our own machines (45 minutes to find “Oh…. this service wasn’t running!”).

We spent quite a bit of time searching the internet for answers on how other businesses were achieving this, and while it seemed nearly every software company we talked to had this problem, most were simply persisting with it being an annoying and time consuming reality of micro-service and infrastructure heavy application development.

At one point in late 2020, we reached out to AWS for their input, who, unsurprisingly, recommended we spin up individual environments for each developer within AWS. Unfortunately, the cost was going to be very high at over $500 per month per developer, as we required things like individual VPCs, API-Gateways and more which cost money even if they’re hardly used. We then investigated creating shared infrastructure for certain things through a hybrid local+cloud development environment, but this was terribly complex. One example of an issue we identified was if a member of our development team was working in our audit service, which listens to SQS, we would need additional infrastructure within AWS that knew when to pipe certain SQS messages to a particular developers machine, vs piping it into the shared, AWS-hosted audit service. Pretty quickly we found the amount of work required to build all of this ugly middleware would derail our teams ability to deliver customer value for months.

It was at this point that our (now) CTO identified Localstack. He was contracting to our company at the time, and as he didn’t have access to any of our AWS infrastructure, was looking for a way to run DynamoDB locally. He had come across Localstack, and briefly explained how it could be used to emulate AWS for local development.

As this was such a big problem for us, we quickly spun up a team to begin figuring out whether we could use Localstack + Docker to run our entire development environment locally.

Our objectives were:

1. Easy to setup for new starters. Ideally 1-click or 1-command.
2. Easy to administer changes and improvements across the whole team. (Very tough when you need all developers to do a bunch of changes to their local environment at the same time to support another change).
3. Cost effective (doesn't add significant cost for each additional headcount)
4. Robust and fault tolerant, we don’t want our development team chasing bugs in their environment.
5. Isolated, so each developer is able to work within their own environment without disrupting others
6. Code which is written in the local environment will work in staging/qa/production without further modification.

Within the first month we had the major pieces of our application up and running within Localstack and Docker (PHP, Databases, Java Services and the SQS/SNS that glued things together). For the Localstack infrastructure, we were able to repurpose the CloudFormation scripts we had built to construct our AWS environments, however some changes were necessary (such as adjusting the way security groups work).

Prior to using Docker, Java Services needed to be compiled on the developers machine, which meant long boot times and lots of software to install locally, even if you were not making changes to any of those services. We updated this to instead download the latest Docker image of each service directly from AWS ECR.

After we had the primary functions up and running, we went onto the more complex ones such as the various serverless applications we run. These were a bit more complex to get working correctly (particularly as they required specific Node versions installed on the developers machine), however we persisted through and got them working reliably.

The Outcome:

When we first switched our developers over to using Localstack and Docker, we encountered a large number of issues that plagued us intermittently for ~6 months as we worked through them all. We had issues with stability (not issues with Localstack's stability, but stability in the way we were using it and our scripts). Certain developers machines would break when setups scripts were altered (while the authors of those changes handled it fine), differences between OSX and WSDL also led a number of issues that needed resolving (our setup guide still contains several steps which are specific to each platform).

There were no silver bullets here, we simply decided to prioritise any local environment issue as high priority and dedicate resources to figuring out how to prevent it from reoccurring.

While it has taken a lot of work to get right, switching to Localstack and Docker has been a huge win for our team and now pays dividends every day. 

Today, a developers entire environment can be booted with a single command `start.sh` in only a few minutes, and the initial environment setup process for a new hire is down to just a few hours (the majority of this time is spent waiting for various things to download that are required by the installation). 

Now, it’s rare that we have a local environment issue, and if we do, we jump on it and get our team moving again. Our team are far more productive and happier, and we can finally say that if the tests are passing locally, they’re going to pass when it goes onto AWS infrastructure.

Issues we encountered and our workarounds:

Hotpotato:

While we were configuring Localstack to receive data from some Lambdas. We were deploying from Serverless and found that Localstack currently (at the time of doing this we were using 12.4, it may have changed since) does not have a way to create a static URL for a Lambda exposed to an API gateway. This was a problem because it meant every time Localstack booted we would get a randomly generated URL and developers would need to update their configurations every time to keep things in line.

So we created a little NodeJS based API proxy which exposes a static URL which will forward the request to a configured Lambda in Localstack, playing hotpotato with the Lambda requests (Hence we called the service “Hotpotato”).

A second very important responsibility was added to this container. Locally, we are not consuming a couple of SQS queues that we do within production, but other services still submit data to them. As nothing is processing these messages, our Localstack ends up accumulating thousands of never to be consumed items. Hotpotato also watches these queues and clears them out as items arrive.

Serverless:

Initially we were using the developers host machine to compile serverless applications prior to pushing into Localstack, but this caused issues with node versioning. Specifically, some serverless apps required a different version of node to each other, and, a different version to our React code base (not ideal, but we simply hadn’t been through upgrading every project to use the latest version of Node). We ended up creating a Docker Container specifically for building our serverless applications and producing a zip file. This container had all the necessary versions of everything pre-installed so our developers no longer needed to manage multiple node versions locally.

Localstack still booting:

Another issue which was particularly challenging to pinpoint, was an issue which intermittently affected most of our Mac developers. The issue would occur “sometimes” when deploying serverless applications, and would manifest itself as SOME of the serverless infrastructure constructing successfully, but other pieces wouldn’t.. It turned out that while we had been able to run our own CloudFormation scripts successfully, some things within Localstack were still booting, and so trying to deploy certain parts of our serverless app would fail.

We ended up checking LocalStack’s Health URL within our start script, and waiting until everything was done being setup:

if command -v /usr/bin/curl >& /dev/null
then
printf "checking for localstack startup\n"
until /usr/bin/curl -s "${LOCALSTACK_URL}/health" --stderr - | grep "\"initScripts\": \"initialized\"" | grep "\"cloudformation\": \"running\"" | grep "\"dynamodb\": \"running\"" | grep -q "\"s3\": \"running\""
do
  echo "Waiting for LocalStack to complete startup..."
  sleep 3
done
printf "${green}Localstack startup verified${reset}\n"
else
printf "curl needs to be installed to check for localstack startup\n"
fi