In the world of Go programming, dealing with concurrent operations is part of the daily routine. As applications grow in complexity, the need to manage and cancel these operations becomes critical. Enter the context package: Go’s solution to managing multiple requests, deadlines, and cancellation signals across API boundaries. This blog post delves into the context package, providing you with the understanding and examples you need to leverage its power in your Go applications.

What is the Context Package

Introduced in Go 1.7, the context package is designed to enable request-scoped values, cancellation signals, and deadlines across API boundaries and between processes. It is particularly useful in applications involving Networking, Infrastructure Components, and Microservices.

Key Concepts

• Cancellation: The ability to signal that an operation should be stopped.
• Deadlines: Setting a time limit on how long an operation should take.
• Values: Storing and retrieving request-scoped data.

Using Context

A context.Context is created for each request by the main function or the middleware of the server. This context is passed down the call chain as a parameter to every function that needs it.

Creating Contexts

The root of any context tree is created with context.Background() or context.TODO(). From there, contexts with deadlines, timeouts, or cancellation signals are derived.

ctx := context.Background()

This context is typically used in main functions, initialization, and tests. It is never canceled, has no values, and has no deadline.

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel() // Important to avoid leaking resources

// Use ctx in operations

This creates a context that automatically cancels after 10 seconds.

Passing Contexts

Contexts are passed as the first parameter of a function. This is a convention in Go programming.

func doSomething(ctx context.Context) error {
    // Function implementation
}

Using Contexts for Cancellation

One of the primary uses of context is to cancel long-running operations. This is crucial for freeing up resources and stopping operations that are no longer needed.

select {
case <-ctx.Done():
    return ctx.Err()
default:
    // proceed with normal operation
}

In this example, we listen for the cancellation signal. If ctx.Done() is closed, we return the cancellation error, effectively stopping the operation.

A practical Example: HTTP Server with Context

Let’s put all this together in a practical example. Imagine we’re building an HTTP server where each request might involve a long-running operation, like querying a database.

package main

import (
    "context"
    "net/http"
    "time"
)

func longRunningOperation(ctx context.Context) (string, error) {
    // Simulate a long-running operation
    select {
    case <-time.After(5 * time.Second):
        return "operation result", nil
    case <-ctx.Done():
        return "", ctx.Err()
    }
}

func handler(w http.ResponseWriter, r *http.Request) {
    ctx, cancel := context.WithTimeout(r.Context(), 10*time.Second)
    defer cancel()

    result, err := longRunningOperation(ctx)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    w.Write([]byte(result))
}

func main() {
    http.HandleFunc("/", handler)
    http.ListenAndServe(":8080", nil)
}

In this example, longRunningOperation listens for a cancellation signal from the context. If the operation takes too long, or if the client disconnects, the operation is cancelled, conserving resources.

HTTP, the foundation of data communication on the World Wide Web, employs a set of methods to define the desired action to be performed on a resource. Among these methods, GET, POST, PUT, and DELETE are fundamental, each serving a distinct purpose. One important concept associated with these methods is idempotency. In this blog post, we’ll delve into these HTTP methods, explore their idempotency, and provide illustrative examples.

HTTP Methods Overview:

1. GET:

The GET method is used to retrieve data from the specified resource. It should only retrieve data and not have any other effect on it at all.

Idempotency: GET requests are inherently idempotent. Repeated identical requests have the same effect as a single request and therefore no side effects whatsoever.

Example:

GET /users/123

2. POST:

POST is used to submit data to be processed to a specified resource. It often causes a change in state or side effects on the server.

Idempotency: POST requests are generally not idempotent. Repeated identical requests may lead to different outcomes, especially if they result in the creation of new resources.

Example:

POST /users Body: { “name”: “John Doe”, “email”: “john@example.com” }

3. PUT:

PUT is employed for updating a resource or creating it if it does not exist at the specified URI.

Idempotency: PUT requests are idempotent. Repeated identical requests should have the same effect as a single request.

Example:

PUT /users/123 Body: { “name”: “Updated Name”, “email”: “updated@example.com” }

4. PATCH:

PATCH is used to apply partial modifications to a resource.

Idempotency: PATCH requests are not guaranteed to be idempotent. Repeated identical requests may or may not have the same effect.

Example:

PATCH /users/123 Body: { “name”: “Updated Name” }

5. DELETE:

DELETE is used to request the removal of a resource at a specified URI.

Idempotency: DELETE requests are idempotent. Repeated identical requests should have the same effect as a single request.

Example:

DELETE /users/123

Understanding Idempotency:

Idempotency, in the context of HTTP methods, means that making a request multiple times produces the same result as making it once. In other words, subsequent identical requests should have no additional side effects beyond the first request.

Key Characteristics of Idempotency:

1. Safety: Idempotent operations should not cause unintended side effects.
2. Predictability: Repeating the same request yields the same result.
3. Statelessness: The server’s state does not change between requests.

Idempotency simplifies error handling and recovery in distributed systems. If a request fails or times out, it can be safely retried without causing unexpected side effects. This property enhances the reliability and robustness of web services.

Idempotency and Retry Mechanism

Retry Strategies in Distributed Systems:

In distributed systems, network issues, temporary failures, or system timeouts are common challenges. When a request fails or times out, retrying the same request can be a natural response to recover from transient errors. However, incorporating retry strategies comes with its own set of complexities, and idempotency plays a crucial role in mitigating potential issues.

How Idempotency Enhances Retry Reliability:

1. Prevents Unintended Side Effects:
   • Idempotent operations ensure that repeating the same request does not result in unintended side effects or duplicate changes in the system.
   • Without idempotency, retries might lead to the execution of the same non-idempotent operation multiple times, causing unexpected alterations in the system state.
2. Consistent State:
   • Idempotent operations provide a guarantee that the system state remains consistent even if a request is retried.
   • Repeated execution of an idempotent operation yields the same result, preventing inconsistencies caused by partial or conflicting updates.
3. Simplifies Error Handling:
   • Idempotency simplifies error handling during retries. Failed requests can be retried without concerns about introducing inconsistencies or undesirable changes.
   • Non-idempotent operations, when retried, may result in varying outcomes, making error recovery more challenging.
4. Enhances Predictability:
   • Idempotency ensures that the outcome of a retried operation is predictable. Developers can rely on the fact that repeating a request will have the same effect as the initial attempt.
   • Predictability is crucial for designing robust and resilient systems, especially in scenarios where network failures or temporary glitches are common.

Implementing Idempotent Retry Strategies:

1. Use Idempotent Operations:
   • Design operations to be idempotent, especially those likely to be retried. This includes operations involving state changes or resource updates.
2. Include Retry Identifiers:
   • Implement mechanisms to include retry identifiers or tokens in requests to deduplicate retries. These identifiers can be used to recognize and discard duplicate attempts.
3. Retry-After Headers:
   • Utilize HTTP Retry-After headers to indicate the recommended time to wait before retrying a failed request. This helps prevent overwhelming the system with repeated immediate retries.
4. Exponential Backoff:
   • Apply exponential backoff strategies to gradually increase the time intervals between retry attempts. This prevents rapid and potentially harmful repeated retries.

Idempotency and Exponential Backoff:

Exponential backoff is a common strategy in retry mechanisms where the waiting time between consecutive retries increases exponentially. Idempotency complements this strategy by ensuring that, regardless of the retry delay, the outcome remains consistent. If the operation is idempotent, the impact of waiting longer before a retry is minimal, as the result will be the same.

Conclusion:

In summary, idempotency and retry mechanisms are intertwined concepts in distributed systems. Idempotent operations provide a foundation for reliable and predictable retries, preventing unintended side effects, maintaining consistent state, and simplifying error recovery. When designing systems that involve retry strategies, incorporating idempotent operations is a key practice for building robust and resilient architectures.

Rob Pike, one of Golang’s dads, 8 years ago gave one of the best programming talks I am aware of at Heroku’s Waza conference titled: “Concurrency is not Parallelism”. The beauty of this talk is his clear compare and contrast exercise between Concurrency and Parallelism whereby Concurrency is defined as:

“Programming as the composition of independently executing processes”

as opposed to Parallelism that is:

“Programming as the simultaneous execution of (possibly related) computations”

Enjoy the video, it is pure gold!

https://www.youtube.com/watch?v=oV9rvDllKEg
Slides: https://go.dev/talks/2012/waza.slide#1

Also the illuminating talk at Google I/0 2012 on Concurrency patter: https://www.youtube.com/watch?v=f6kdp27TYZs

At work, I was recently introduced to the idea of Monorepos. While developer who have been writing code for microservices tend to think of microservice architecture as a number of x codebases somehow interacting with each other, there is another school of thoughts that is pushing the idea of organising code in a single repository. Do not fret, that does not mean that we want to create a monolithic application, tightly coupled, aka a giant and far from being modular piece of software. It only means that, instead of having the code residing in different repositories and having to tweak and make changes in each of one making sure that integration tests are still green and happy, everything is in one codebase and it generally holds the following features:

• Only build the services or cmds that are modified in a pushed commit;
• Build all services and/or cmds that are affected by changes in common codes (i.e. pkg);
• Build all services and/or cmds that are affected by changes in vendor codes.

https://medium.com/goc0de/how-to-golang-monorepo-4f62320a01fd

https://circleci.com/blog/monorepo-dev-practices/

https://hardyantz.medium.com/getting-started-monorepo-golang-application-with-bazel-370ed1069b4f

ApacheCon, the global Apache conference, has arrived and it landed in your own very living room. That is possibly one of the few times in the past year and half where you might find yourself saying “Thank you COVID!”. ApacheCon is not only fully online, but it is also free (unless you want to donate to Apache Foundation, which is all well and good). Here is the link to register:

https://www.apachecon.com/acah2021/

Have fun!

https://link.medium.com/4FxI4bN7Bcb

A while ago, I was tasked with writing a bash script that would do the following:

• Read the username, password and host values from Kubernetic config map
• Reads the Postgres username, password, host and database values
• Replace variables with corresponding values in the application.yml file

#! /usr/bin/env sh

echo "Mapping the env variables to the values from Kubernetis "

STAGING_ENV="$1"

#Sanity check
if [ $# -eq 0 ]; then
    echo "No argument was provided, however the script requires 1 argument to successfully run"
    exit 1
fi

POSTGRES_CORE="postgres-core"

#Postgres Env Variables
export DB_USERNAME=$(kubectl get configmap "$STAGING_ENV"-$POSTGRES_CORE -o jsonpath="{.data.username}")
export DB_PASSWORD=$(kubectl get secret "$STAGING_ENV"-$POSTGRES_CORE -o jsonpath="{.data.password}" | base64 -D)
export DB_HOSTNAME=$(kubectl get configmap "$STAGING_ENV"-$POSTGRES_CORE -o jsonpath="{.data.host}")
export DB_NAME=$(kubectl get configmap "$STAGING_ENV"-$POSTGRES_CORE -o jsonpath="{.data.name}")

export APPLICATION_YML_LOCATION=src/main/resources/application.yml

echo "Start replacing postgres env variables values with sed"
sed -i '' "s/DB_USERNAME:postgres/DB_USERNAME:$DB_USERNAME/g" $APPLICATION_YML_LOCATION
sed -i '' "s/DB_PASSWORD:password/DB_PASSWORD:$DB_PASSWORD/g" $APPLICATION_YML_LOCATION
sed -i '' "s/DB_HOSTNAME:localhost/DB_HOSTNAME:$DB_HOSTNAME/g" $APPLICATION_YML_LOCATION
sed -i '' "s/DB_NAME:core-db/$DB_NAME:$DB_NAME/g" $APPLICATION_YML_LOCATION
echo "End replacing postgres env variables values with sed"

Source: New from Satellite 2020: GitHub Discussions, Codespaces, securing code in private repositories, and more – The GitHub Blog