# GitHub Demo Project

This blog post references this [demo project](https://github.com/Compulsed/cdk-lambda-typescript-otel-honeycomb). You can deploy this project to your AWS environment to see everything up and running.

![Example Trace](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/c5839fab-5ab1-43b0-9389-d4519bb9cde1-Screenshot%202022-12-11%20at%204.56.56%20PM.png)

After running the above project you should see a trace like the above.

# The Problem

**Why can I not just import and use @opentelemetry/api?**

You can. Though, the issue with this package is that it buffers traces in memory and then periodically pushes them to the telemetry service. This buffering creates issues with AWS lambda. This problem is because once the handler is run, the Lambda Service will [freeze](https://www.youtube.com/watch?v=yDplzXEdBTI) the container. Because the Lambda container is frozen, the traces may never end up at the telemetry service.

# The solution

AWS Attempted to solve this problem with Lambda [Extensions](https://docs.aws.amazon.com/lambda/latest/dg/lambda-extensions.html). These extensions run in the background as a sort of 'sidecar'. This sidecar can continue to run and push metrics to an OTEL service even after the hander has finished. 🎉

The AWS maintained extension is this [aws-observability/aws-otel-lambda](https://github.com/aws-observability/aws-otel-lambda). This extension embeds the work done here [open-telemetry/opentelemetry-lambda](https://github.com/open-telemetry/opentelemetry-lambda). Contained with the `aws-otel-lambda` package is a version of `@opentelemetry/api`, we will reference this package from within our lambda function.

## If only it were that simple

Getting the above solution working with the TypeScript CDK is difficult, to say the least. It requires a bunch of changes to the default CDK project.

### 1. Setting up the layer

Below is the baseline configuration you will need to get the layer working/deployed. The lambda extension uses the `collector.yaml` to determine how it should be set up. The `commandHooks` instructs ESBuild to include the file lambda zip.

*File: lib/hello-world.ts*
~~~ts
// From https://github.com/aws-observability/aws-otel-lambda
layers: [
  lambda.LayerVersion.fromLayerVersionArn(
    this,
    'otel-layer',
    'arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-arm64-ver-1-7-0:2' // Replace the region based on where the lambda is deployed
  ),
]
~~~

*File: lib/hello-world.ts*
~~~ts
environment: {
  AWS_LAMBDA_EXEC_WRAPPER: '/opt/otel-handler',
  OPENTELEMETRY_COLLECTOR_CONFIG_FILE: '/var/task/collector.yaml',
  OTEL_PROPAGATORS: 'tracecontext',
  OTEL_SERVICE_NAME: 'demo-example-service',
}
~~~

*File: lib/hello-world.ts*
~~~ts
commandHooks: {
  beforeBundling(inputDir: string, outputDir: string): string[] {
    return [`cp ${inputDir}/collector.yaml ${outputDir}`]
  },
  afterBundling(): string[] {
    return []
  },
  beforeInstall() {
    return []
  },
}
~~~

*File: collector.yaml*
~~~yaml
receivers:
  otlp:
    protocols:
      grpc:
      http: 

exporters:
  otlp/traces:
    endpoint: api.honeycomb.io:443
    headers:
      'x-honeycomb-team': xxx

  logging:
    loglevel: debug

service:
  extensions: []
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [otlp/traces]
~~~

### 2. Lack of support for ESM Modules for handler exports

Normally you would export a lambda function with `const export handler = () => {}`. This is not possible due to [open-telemetry/opentelemetry-js/issues/1946](https://github.com/open-telemetry/opentelemetry-js/issues/1946). This appears to be an issue with the way that the auto instrumentation wants to monkeypatch the handler. The way we can get around this is to simply export it via module exports.

*File: lib/hello-world.function.ts*
~~~ts
module.exports = { handler }
~~~ 

### 3. Referencing the NodeJS OTEL library in the layer / Lack of support for ESM Modules for @opentelemtry imports

Normally you would be able to just import `'@opentelemetry/api'` by running `import { trace, SpanStatusCode }  from '@opentelemetry/api'`. This appears to be an issue with the lambda layer. The solution to this is just using `require`, then import the types.

*File: lib/hello-world.function.ts*
~~~ts
const { trace, SpanStatusCode } = require('@opentelemetry/api')
import type { Span, SpanOptions } from '@opentelemetry/api'
~~~

Also, because we want to reference the `@opentelemetry/api` from our layer we need to exclude it from the bundling step.

*File: lib/hello-world.ts*
~~~ts
externalModules: [
   '@opentelemetry/api',
   '@opentelemetry/sdk-node',
   '@opentelemetry/auto-instrumentations-node',
]
~~~

### 4. Root Span & XRay Tracing

AWS has native XRay integration with Lambda. This means that when your function gets invoked, it has X-Ray information from the Lambda service that is passed into it. This means that when we work with OTEL, it will already have a parent span (the lambda service). The problem is that span will not go to Honeycomb, and Honeycomb will think there is a missing span. To fix this issue we just say that our first span is our root span.

To fix this we need both of these options:

*File: lib/hello-world.ts*
~~~ts
tracing: lambda.Tracing.PASS_THROUGH
~~~

*File: lib/hello-world.function.ts*
~~~ts
tracer.startActiveSpan('handler', { root: true }, async (span: any) => {
   // your code here
}
~~~

### 5. ESBuild Bundling, Auto instrumentation

The TypeScript CDK uses ESBuild to compile and build your lambda function. ESBuild will attempt to pull all of your node module dependencies and source dependencies into a single file. This helps to improve lambda cold starts.

The problem with this bundling is that it messes with OTEL's [auto-instrumation](https://www.npmjs.com/package/@opentelemetry/auto-instrumentations-node). To get the auto-instrumentation working again, you need to specify each package in the `nodeModules` section of ESBuild. This option stops ESBuild from trying to bundle them together and just keeps those dependencies in the node_modules folder.

*lib/hello-world.ts*
~~~ts
nodeModules: [
  '@aws-sdk/client-dynamodb',
],
~~~

# Summary

AWS Lambda, Honeycomb, and OTel are great services. They are even better together. It took me a long time to put this together so let me know if it was helpful

Instrumenting a Typescript Lambda function with Honeycomb / CDK is hard. This guide will run you through how you configure your Lambda function so that all your traces / spans end up at Honecyomb.

Instrumenting a Typescript Lambda function with Honeycomb / CDK

🐝 Typescript CDK Lambda Function with OTEL / Honeycomb

# Podcast

You can find Yan Cui on twitter: [@theburningmonk](https://twitter.com/theburningmonk)

Link to the postcast: [Postcast](https://www.buzzsprout.com/877747/5009156) 

A Cloud Guru is a company that started out Serverless from the start. In this podcast Yan Cui and I talk about:

- AppSync
- GraphQL
- Monorepos & sharing packages
- Scaling teams
- Microservice communication
- Thick clients
- Firebase

Discussion of the evolution and some of the challenges of being a completely serverless company

🎙 Podcast with Yan Cui on A Cloud Guru's Serverless Journey

A Cloud Guru has been an entirely serverless company from day one. 

The reason that Sam Kroonenburg chose serverless when building A Cloud Guru was to reduce the time to market. Five years later, serverless has been an incredible productivity booster for the organization. It’s enabled the dream of small and lean product teams delivering features to production multiple times per day. Serverless architecture means that these teams can spend less time managing infrastructure and more time building features.

But, as they say, there’s no such thing as a free lunch. In the case of serverless, that’s meant we have specifically had to architect our application to solve the performance issues that can come with it. The way we improved performance? Minimizing the impact of cold starts and the communication between lambda functions. Here’s how.

# How we reduced the impact of Lambda cold starts and improved our lead times with GraphQL Schema Stitching

What’s a [cold start](https://mikhail.io/serverless/coldstarts/aws/)? Here is an excellent explanation.

<img style="width: 50%" src="https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/7dd80251-c0d1-4e1e-812f-fce8cc30af02-performance-matters.webp" />

*Feedback from students that performance is important*

When it comes to managing the impact of cold starts, the first place to look is the request-response APIs that are driving frontend behaviour. In the case of A Cloud Guru, our API layer was written in GraphQL.

The slow responsiveness was the main reason we had looked into improving our GraphQL layer. As a secondary point, we wanted to improve developer happiness. This could have been done by reducing time to production ([lead times](https://clubhouse.io/blog/lead-time-what-is-it-and-why-should-you-care/)). For common changes like extending or deprecating an API, we didn’t want for developers to make changes in two places, which could require additional deployments and code reviews for each step. 

Let’s run through the evolution of our architecture and how those changes reduced the performance issues and improved our lead time.

# A GraphQL primer using web series

The main appeal of GraphQL is to fetch all the data needed for a single page within one request. This decision was particularly important because we were also building our first mobile application at the same time. For GraphQL to be able fetch data within one request, there has to be a single endpoint that’s capable of resolving the entire application data graph. That entry point is our gateway. Our gateway is just used as a single point of entry and by design doesn’t contain business logic.

![API](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/e0ca6da8-a727-42b2-a20a-36c627d91ef5-xray-screen.webp)

A typical request that would be used for loading our web series page. For example, this landing page.

![System Architecture](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/df25817a-23dc-43f0-9c41-b6f2e3f64b54-system-architecture.webp)

*Systems architecture / entity association of microservices involved in resolving web series.*

GraphQL is just a specification on the format of a query. It doesn’t prescribe some key details: **where the schema definition should exist, how the data is resolved within a microservice (intra-domain), and how data is resolved between a service (inter-domain)**. Two different architecture approaches adjust these two variables.

# Lambda functions invoked by a Graphql gateway

[Link to Code](https://github.com/ACloudGuru/blog-acg-graphql-evolution/tree/master/01-gateway-invoke)

**Location of schema definition**: Located on the gateway

**Data resolution**: Gateway performs Lambda Invokes into the Microservice to resolve data; gateway owns intra and inter type connections.

![Service to service](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/f2541bf1-3fea-4a82-b4ff-5c6ce808dd21-service-to-service.webp)

*Service to service was managed through AWS SDK Lambda Invoke.*

This architectural approach just used GraphQL.js on the gateway. This gateway would own the entire schema and manage the resolution of queries into our backend. That way the gateway would reach out into each microservice using the AWS SDK’s Lambda Invoke.

~~~js
const response = await lambda
   .invoke({ FunctionName: 'series-dev-allSeries' })
   .promise();

return getInvokeBody(response);
~~~

*An example AWS Lambda Invoke call the the gateway would make. [Reference](https://github.com/ACloudGuru/blog-acg-graphql-evolution/blob/7cff1d84c9a88356a633abefea0b6379b6eddee7/01-gateway-invoke/gateway/src/handler.js#L57)*

![Data Resolution](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/2a7e285e-f72c-4cbb-94a6-1db417b57abb-data-resolution.webp)

*Data resolution for query is orchestrated by the gateway.*

The resolution of the data is as follows:

The gateway resolves all the series

- When the series is known, the gateway then asks for all the episodes within the series
- Due to the episodes entity owning access permissions, it has the business logic on whether someone should see the episode or not
- Once the resolution of the episodes happens, only then can the gateway resolve the author from our identity service

![Sequence diagram](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/118db93b-f604-4dab-b6d6-9c9ca616ce38-sequence-diagram.webp)

*A sequence diagram illustrated with XRay. Series + Identity called on gateway, and content called on series.*

XRay illustrates two key things: the time it takes to initialize a cold lambda function, and how long it takes to fetch the data. The initialization only has to happen once per lambda container, as the same container is re-used for subsequent requests (until scaling up or the container gets recycled).

What we noticed with this query is that it crosses **five lambda functions to resolve all of the required data**. There are two types of lambda functions: lambda functions that are invoked frequently (gateway lambda) and lambda functions that may not be (2x Series, 1x Content, 1x Identity).

The example above is the best-case example for the lambda functions. (They are light on code dependencies because code dependencies introduce longer cold starts.) In the typical example, most lambda cold starts are around one second. This means a user hitting the page where four lambdas aren’t warm (haven’t been invoked in 15 minutes) could be experiencing a five-second delay. Ouch.

There are a couple of other problems with this architectural approach:

- The GraphQL schema exists on the gateway. This requires two deployments when changing an API — increasing lead time and decreasing developer happiness
- Communication is coupled to Lambda, meaning we wouldn’t be able to have microservices that don’t use lambda functions. (This turned out to be important when integrating Labs and Playgrounds from the Linux Academy Platform.)

# Schema stitching

[Link to Code](https://github.com/ACloudGuru/blog-acg-graphql-evolution/tree/master/02-schema-stitching)

**Location of schema definition**: In each microservice

**Data resolution**: Gateway delegates query resolution to microservices; gateway owns inter domain type connections; microservices own intra domain type connections.

![Gateway Comms](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/814e33e0-b3bd-42d7-ad2f-9662d1ebddfd-gateway-comms.webp)

*Now communicating from the gateway to the downstream services using schema delegation / stitching. RPCs also managed through connecting via GraphQL.*

~~~js
const mergedSchemas = stitchSchemas({
    subschemas: [
        seriesSchema,
        contentSchema,
        identitySchema    
    ],
    typeDefs: `
        extend type Episode {
            author: User
        }
    `,
    resolvers: {
        Episode: {
            author: {
                selectionSet: `{ authorId }`,
                resolve(episode, _args, context, info) {
                return batchDelegateToSchema({
                    schema: identitySchema,
                    operation: 'query',
                    fieldName: 'usersByIds',
                    key: episode.authorId,
                    argsFromKeys: (ids) => ({ userIds: ids }),
                    context,
                    info,
                });
                },
            },
        }
    }
});

const response = await graphql(
    mergedSchemas,
    payload.query
);
~~~

*Stitching the schemas together and doing a cross domain link between episode and author entities together on the gateway. [Reference](https://github.com/ACloudGuru/blog-acg-graphql-evolution/blob/7cff1d84c9a88356a633abefea0b6379b6eddee7/02-schema-stitching/gateway/src/handler.js#L63)*

![Delegate](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/e46c8cb6-1b7f-450f-94b4-0e44ee626698-delegate-sub-queries.webp)

*We delegate sub queries to responsible microservices*

The resolution of the data is as follows:

- The gateway delegates the web series portion of the query to the series service
- Due to the episodes entity owning access permissions, it has the business logic on whether someone should see the episode or not, so it still manages the resolution of the content service
- Once the resolution of the episodes happens, only then can the gateway delegate the resolution of authors through the identity service

![Sequence Diagram](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/91e8852a-61ae-4aac-90ab-cacaf69cebcf-sequence-diagram-2.webp)

Sequence diagram illustrated with XRay. Series + Identity called on gateway; content called on series. Cold-start initialization omitted due to the likeliness of the lambda being warm.

Using this approach, each microservice’s API is now represented as a single lambda function. This does two things:

- Intra domain connections (e.g. series + episodes resolution happens within the same lambda function rather than across two). This reduces the number of possible starts + service to service invocation time
- The lambdas are likely going to be warmer because there is only one lambda function for an entire service

We have transitioned from five possible cold starts down to four (gateway, series + episode, content, author), **though each lambda is less likely to have a cold start** as the API lambda is handling lots of different operations for that microservice.

Transitioning from the gateway owning the schema to schema stitching has solved some of the issues in the first approach:

- Changes to the API and service can be done in one deployment, improving our lead time
- No longer coupled to AWS lambda as we are using HTTP to communicate

This approach did have some problems, though it came with some trade-offs:

The gateway still had to manage inter domain connections (albeit, less likely to be in comparison to intra connections)

- Additional latency was added by using API Gateway (could have technically still been done with Lambda Invoke)
- Security is managed by x-api-keys rather than IAM

Monitoring became harder because there is one GraphQL lambda for all the API requests a service handles. We had decided to use a GraphQL specific service to solve this problem (Apollo Engine). Adding an additional service does mean that we now have multiple places where you can search for issues with API requests.

For the gateway to know how to delegate the request microservice, we have to query every microservice. (At this point we had over 40.) Querying service microservice would introduce either a longer cold start OR cache invalidation problems. We ended up solving this problem by adding in-house service discovery. Every time a request came in, the gateway would make a single request to the service discovery tool to get the entire schema, url and api keys of each microservice. Note: Service discovery is not included in the example code, adding three additional introspection calls.

# Cold start analysis

To compare the different approaches, we used ApacheBench and then exported the results into Excel. To capture the effects of cold starts, all functions were re-deployed before. A consideration is that this ApacheBench test is completely synthetic (all at once) and not indicative of real-world traffic (staggered over the day).

~~~sh
# Force coldstarts
./deploy.sh

# Run tests
ab \
    -n 1000 \
    -c 5 \
    -T "application/json" \
    -p "post-request.json" \
    -g "gateway-invoke-results.tsv" \
    "https://11gx93bhx8.execute-api.us-east-1.amazonaws.com/dev/graphql"
~~~

ApacheBench running performance testing [Reference](https://github.com/ACloudGuru/blog-acg-graphql-evolution/blob/master/01-gateway-invoke/performance.sh)

![vs](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/05cc11e1-4964-4cf6-aa89-a7a4f35ac532-vs.webp)

Gateway Invoke vs. Schema Stitching [Raw Results](https://github.com/ACloudGuru/blog-acg-graphql-evolution/blob/master/Results.xlsx)

There appear to be two key differences in the results of the data.

In this specific query schema, stitching has a 20% lower p99 (five seconds to four seconds). The lower p99 is due to one less lambda function invocation / cold start when resolving the series data. What can’t be easily shown in synthetic tests is the reduction in cold starts due to the lambda functions being kept warm by other calls made to these microservices. We would suspect there’s a higher utilization of that single API lambda function compared to the many lambda functions that the gateway approach has. This would mean significantly fewer cold starts over time.

Gateway Invoke performance is better between the p60-p70 mark. In the schema stitching approach, we introduced API Gateway when communicating between microservices. It appears that adding API Gateway adds additional variance to the results. If we had decided to continue to use lambda invoke for schema stitching rather than API Gateway, this might have reduced that inconsistency.

# Lead time analysis

It is a little harder to pin-point the exact amount of API changes that are happening to our gateway. Though, we can do some rough calculations by using data pulled from our Apollo Studio changelog. We can see that in the Month of September, we had roughly 230 changes to our GraphQL schema.

![Apollo Studio](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/c0ce70a6-b364-417d-8190-0646c7dbc344-apollo-studio.webp)

*Apollo Studio calculating GraphQL schema diff*

By looking at what’s contained within the diff, it would appear that there are roughly 20 deployments that have a schema change within them. It wouldn’t unreasonable to think a schema change could take around three engineering hours, a PR (context switch for the other engineer reviewing the code) and to deploy the gateway. That means we could have potentially saved 60 hours per month by bundling and deploying schema changes along with code changes.

# Conclusion

With the goal to improve the overall student experience, we’ve made some significant architectural changes.

Schema stitching allowed for fewer cold starts, and when they did happen, they would have a lower impact (at least 20%). Moving to schema stitch also had a large reduction in the lead time to get an API change from development to production (across all our engineers, roughly 60 hours per month).

A Cloud Guru moved from having multiple lambdas in a microservice to a single API Lambda function. This Lambda function was federated together via GraphQL Federation.

A Cloud Guru reduced Lambda cold starts/lead time for changes using GraphQL Federation

🥶 How we reduced Lambda cold starts at ACG

# Video
A Cloud Guru has been a Serverless company from day one. Being a Serverless company does not necessarily mean that we started with all the patterns that would effectively allow us to scale out our development team. 

This talk is a brief history and analysis of our original architecture, deployment strategy, service communication. I then contrast that history against our new ideas and approaches to building Serverless architectures.

**NOTE:** these slides were missing from the video: [PDF](https://bloginfrastructure-prod-imagebucket97210811-1mgqvezvs9c8h.s3-accelerate.amazonaws.com/87413ff6-63c0-4f77-988b-d2445478cd8a-ServerlessDays.pdf)</span>

<br />

<div style="position: relative; width: 100%; height: 0; padding-bottom: 56.25%;">
     <iframe src="https://www.youtube.com/embed/WDd2qfEwdzQ" frameborder="0" allowfullscreen style="position: absolute; top: 0; left: 0; width: 100%; height: 100%;"></iframe>
</div>


A Cloud Guru's serverless microservice architecture

🦘 Serverless Days Sydney 2019 (Video)

# Serverless: What, how and why?

More people are beginning to see serverless as the future of software architecture. And why not? There is a significant amount of appeal in the idea of developing an application without having to worry about the servers.

Those of you who still have no idea what serverless is, or what it can do – don’t worry. Despite the hype, it’s still a growing trend, and there’s time to get into it via the many meetups and workshops that pop up in cities around the world.

Here’s a basic guide I’ve put together to help answer some common questions about serverless.

## What does serverless really mean?

The term “serverless” is a bit of a misnomer. Serverless isn’t 100% serverless, or even close. There are still servers, virtual machines, operating systems, containers, and runtimes, you just never have to manage them.

When people refer to serverless computing casually, most of the time they are referring to Functions as a Service (FaaS), and a handful of other 3rd party services which provide either data persistence or data transformation features.

The FaaS part of serverless computing is built on top of all existing and conventional abstraction layers. In the case of AWS Lambda (the most widely known FaaS option), it goes like this:

1. Physical hardware runs a virtual machine (EC2).
2. The virtual machine then runs an operating system (such as Linux).
3. The operating system runs a container (Firecracker). Containers are a way of standardising application packaging, making it easy to operate an application without dealing with ordinary incompatibility problems.
4. The container runs a runtime, which then starts executing your code.

During this process, a whole lot of other load-balancing and scaling magic happens in the Lambda control plane, but you never see or need to deal with those complexities.

## But wait, what about cloud computing?

You might have heard of cloud computing spoken about alongside serverless. Cloud computing isn’t the same as serverless. It is the platform and abstraction in which serverless computing is achieved.

Cloud computing is a generic term describing an abstraction over remote physical hardware that is controlled through the use of software. The term cloud computing does not describe specifically how people manage, organise, and control those remote computing resources, or even specifically what they are or do.

When people describe cloud computing, they typically refer to a common set of attributes that are easily and commonly architected for in-cloud native solutions. These include, but are not limited to:

- On-demand resources
- Pay for what is used
- Elasticity
- Economy of scale
- Software-defined infrastructure

## What are the benefits of serverless computing?

Serverless maximizes all the benefits of the cloud and makes attributes that are desirable in well-architected software systems super easy to achieve.

The benefits of serverless computing are the benefits of the cloud, just without having to deal with the typical architectural challenges of designing for them. These benefits include:

- High availability
- Scalability
- No operational overhead for managing concerns below your layer
- Serverless is immutable
- Encouraged statelessness at the architectural and code level
- Cloud-native service integrations
- A completely code-defined environment


Dale Salter

Serverless, Software Engineering, Leadership, DevOps