Building A Custom Payroll System: MongoDB, Go, and Yellowcard API Series - Part 1
Simplifying MongoDB in Go: Building a Payroll System Tutorial
It took an entire day to decide on a project for an assessment test. I kept going back and forth between my flatmate's room and mine, explaining the benefits of building a currency rate tracker instead of a payroll system to showcase my technical skills.
Well, I finally decided to build a payroll system using MongoDB, which wasn't commonly used by the Go community, so library support was limited compared to SQL π« . Another decision I had to make was how to build the application. Having a sketch on paper is enough to set up the application code. Choosing the right routing library is also important. In this case, we will use the Gin library framework.
Let's get started.
Architecture ποΈ
I didn't plan to discuss this application's architecture right away. I wanted to explain it while going through the code, but one of my reviewers requested it, so let's dive in.
Design decisions, thought processes, trade-offs, and pros and cons are crucial when building an application. For the payroll system, MongoDB was chosen to explore the synergy between Golang and MongoDB and avoid migrations like in SQL, especially since all assessments come with deadlines.
Let's take a peek at the application design diagram for a while.
Here is a rough illustration of the application. Initially, I tried supporting database pools, but for simplicity's sake and to avoid premature optimization, I removed it. Instead, I focused on the following assumption.
Businesses can add, edit, delete employees and make payments.
All payments are in Naira only
Refunds and cancellations are not supported
The account has been funded via the YellowCard dashboard
The user is the business owner
The provided information has been validated
KYC meta information for employees and employers has been collected and verified the above assumptions and requirements help determine the key modules to focus on: the employee, webhook, authentication, and disbursement modules.
That's about it for the assumptions of this design. We will explore more as we move forward.
Configuration π
Usually, every application starts with identifying configuration properties. One important thing to note is that we won't be using an env file but a yaml file for all our configurations. However, the code includes support for loading from an env file.
Let's have a closer look at it. Create a file common/config.go
and add some structures for storing environmental file paths.
type ConfEnvSetting struct {
YamlFilePath []string
EnvFilePath string
}
This structure is designed to store the paths to all the files we want to load. Next, we add our application configuration details
type Config struct {
LogLevel string `config:"logLevel"`
ServerPort string `config:"serverPort"`
YellowCardCredentials struct {
ApiKey string `config:"apiKey"`
SecretKey string `config:"secretKey"`
BaseUrl string `config:"baseUrl"`
}
JWTCredentials struct {
AccessTokenSecret string `config:"accessTokenSecret"`
AccessTokenClaim struct {
Issuer string `config:"issuer"`
Audience string `config:"audience"`
}
AccessTokenTTL time.Duration `config:"accessTokenTTL"`
}
MongoDB struct {
DBUri string `config:"dbUri"`
DatabaseName string `config:"databaseName"`
}
AllowedCorsOrigin []string `config:"allowedCorsOrigin"`
}
Still in common/config.go
, let's run go mod init <your-project-name>
to set up our go.mod
and go.sum
files to manage external library dependencies. In this case, we will install the following libraries: github.com/gookit/config/v2
, github.com/gookit/config/v2/yaml
, and github.com/samber/lo
.
Now, let's add the LoadConfiguration
function to load the configuration into our main.go
.
import (
"encoding/json"
"fmt"
"log"
"sync"
"time"
"github.com/gookit/config/v2"
"github.com/gookit/config/v2/yaml"
"github.com/samber/lo"
)
// Insert the previous code hear.
var (
cfg = &Config{}
loaded = false
)
var once sync.Once
func LoadConfiguration(cfgEnvSetting ConfEnvSetting) (*Config, error) {
if loaded {
return cfg, nil
}
config.WithOptions(func(opt *config.Options) {
opt.DecoderConfig.TagName = "config"
})
config.WithOptions(config.ParseEnv)
if len(cfgEnvSetting.YamlFilePath) != 0 {
config.AddDriver(yaml.Driver)
}
files := []string{}
if !lo.IsEmpty(cfgEnvSetting.EnvFilePath) {
files = append(files, cfgEnvSetting.EnvFilePath)
}
files = append(files, cfgEnvSetting.YamlFilePath...)
err := config.LoadFiles(files...)
if err != nil {
return nil, err
}
once.Do(func() {
err := config.Decode(cfg)
if err != nil {
log.Panic(err)
}
loaded = true
})
if cfg.LogLevel == "debug" {
cfgJson, err := json.MarshalIndent(cfg, "", " ")
if err == nil {
fmt.Printf("Cudium-Backend API Config: %s\n", cfgJson)
}
}
return cfg, nil
}
func GetConfig() *Config {
return cfg
}
With this done, let's set up our logger.
Logging πͺ΅
Logging should be a priority for any application being built as it helps identify where issues might occur, especially when dealing with bugs in production π.
From my personal experience, using logged stack traces is very important to easily identify where crashes or bugs might lie. This helps pinpoint the line that caused the error.
Let's see how we can implement a basic logging package for the application.
Create an internals/logger.go
file and put the code below in it. I will explain what the code does in a bit.
package internals
import (
"os"
"github.com/op/go-logging"
)
// Logger is a server logger
type Logger interface {
Fatal(args ...interface{})
Fatalf(format string, args ...interface{})
Panic(args ...interface{})
Panicf(format string, args ...interface{})
Critical(args ...interface{})
Criticalf(format string, args ...interface{})
Error(args ...interface{})
Errorf(format string, args ...interface{})
Warning(args ...interface{})
Warningf(format string, args ...interface{})
Notice(args ...interface{})
Noticef(format string, args ...interface{})
Info(args ...interface{})
Infof(format string, args ...interface{})
Debug(args ...interface{})
Debugf(format string, args ...interface{})
}
// LoggerID ia a logger ID
const LoggerID = "yc-backend"
var (
// Logger settings
logger = logging.MustGetLogger(LoggerID)
logConsoleFormat = logging.MustStringFormatter(
`%{color}%{time:2006/01/02 15:04:05} [YC-Backend] >> %{message} %{color:reset}`,
)
)
func init() {
// Prepare logger
logConsoleBackend := logging.NewLogBackend(os.Stderr, "", 0)
logConsolePrettyBackend := logging.NewBackendFormatter(logConsoleBackend, logConsoleFormat)
lvl := logging.AddModuleLevel(logConsolePrettyBackend)
logging.SetBackend(lvl)
//logging.SetLevel(logging.INFO, LoggerID)
// Set log level based on env
switch os.Getenv("APP_LOG_LEVEL") {
case "debug":
logging.SetLevel(logging.DEBUG, LoggerID)
case "info":
logging.SetLevel(logging.INFO, LoggerID)
case "warn":
logging.SetLevel(logging.WARNING, LoggerID)
case "err":
logging.SetLevel(logging.ERROR, LoggerID)
default:
logging.SetLevel(logging.INFO, LoggerID) // log everything by default
}
}
// GetLogger returns the logger
func GetLogger() Logger {
return logger
}
In this code snippet, we created a logger interface to help us print application runtime information to the console.
Next, let's build our application engine.
Engine π
You might be wondering, "Why do we need an engine?" We need a configurable engine to bring together all the necessary parts to make the application work. For example, if our application requires a database, this is the perfect place to integrate it, along with any other components we need to add.
Let's create a folder engine/app.go
and add the following line of code to the file. Also, import the following libraries github.com/gin-contrib/cors
,github.com/gin-contrib/gzip
,github.com/gin-gonic/gin
,github.com/samber/lo
and go.mongodb.org/mongo-driver/mongo
.
import (
"context"
"errors"
"log"
"net/http"
"os"
"os/signal"
"sync"
"syscall"
"time"
"yc-backend/internals" // import the logger
"github.com/gin-contrib/cors"
"github.com/gin-contrib/gzip"
"github.com/gin-gonic/gin"
"github.com/samber/lo"
"go.mongodb.org/mongo-driver/mongo"
)
type Application struct {
Config *common.Config
DB *mongo.Client
Logger internals.Logger
Context context.Context
server *http.Server
mux *gin.Engine
wg sync.WaitGroup
quit chan os.Signal
}
func (srv *Application) Setup() *Application {
r := gin.New()
// we will add middleware integration ...
r.Use(gzip.Gzip(gzip.DefaultCompression))
r.Use(gin.Logger())
r.Use(gin.Recovery())
r.Use(func(ctx *gin.Context) {
cfg := common.ConfigFromCtx(ctx)
if cfg == nil {
panic(errors.New("allowed cross-origin not set"))
}
cors.New(cors.Config{
AllowMethods: []string{"GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"},
AllowHeaders: []string{"Accept", "Authorization", "Content-Type", "X-CSRF-Token"},
AllowCredentials: true,
ExposeHeaders: []string{"Content-Length"},
AllowOriginFunc: func(origin string) bool {
return lo.Contains(cfg.AllowedCorsOrigin, origin)
},
MaxAge: 30 * time.Second,
})(ctx)
ctx.Next()
})
r.GET("/status-check", func(ctx *gin.Context) {
ctx.JSON(http.StatusOK, utils.SuccessResponse("aliveπ«΅", gin.H{"status": "ok"}))
})
srv.mux = r
srv.wg = sync.WaitGroup{}
srv.server = &http.Server{
Addr: srv.Config.ServerPort,
Handler: srv.mux,
ReadTimeout: 15 * time.Millisecond,
WriteTimeout: 30 * time.Millisecond,
ReadHeaderTimeout: 15 * time.Millisecond,
}
srv.quit = make(chan os.Signal, 1)
signal.Notify(srv.quit, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)
return srv
}
In the snippet above, we created an application structure, attached an HTTP server, and configured our router. A status check endpoint lets us easily verify if our server is active.
Let's handle graceful shutdown and server listener logic also for our base logic
func (srv *Application) GracefulShutdown() {
go func(quit chan os.Signal, dbm *mongo.Client) {
<-quit
shutdownCtx, shutdownCancelFunc := context.WithTimeout(srv.Context, 5*time.Second)
go func() {
defer shutdownCancelFunc()
<-shutdownCtx.Done()
if shutdownCtx.Err() == context.DeadlineExceeded {
log.Fatal("graceful shutdown timed out.. forcing exit.")
}
}()
err := srv.server.Shutdown(shutdownCtx)
if err != nil {
log.Fatal(err)
}
err = dbm.Disconnect(shutdownCtx)
if err != nil {
log.Fatal(err)
}
log.Fatal("graceful shutdown... forcing exit.")
}(srv.quit, srv.DB)
}
func (srv *Application) ListenAndServe() error {
if err := srv.mux.Run(srv.Config.ServerPort); err != nil {
return err
}
<-srv.Context.Done()
return nil
}
We expect that when we try to shut down the server, all currently handled requests will be completed successfully. This code should also be included in the engine/app.go
file.
Database Setup π
For the final part of this article, which will be part of a possible 3-part series, we will set up main.go
in our root folder where our go.mod
the file is located. Then, add the following code to connect our MongoDB database to our application. In the next parts of the series, we will learn how to use the repository model to work with the Mongo client.
import (
"context"
"log"
"time"
"github.com/gin-gonic/gin"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
func main() {
// code goes here...
}
func setupDatabase(clientOpts *options.ClientOptions) *mongo.Client {
ctx, cancelFunc := context.WithTimeout(context.Background(), 15*time.Second)
defer cancelFunc()
client, err := mongo.Connect(ctx, clientOpts)
if err != nil {
log.Fatal(err)
}
ctx, cancelFunc = context.WithTimeout(context.Background(), 1*time.Second)
defer cancelFunc()
err = client.Ping(ctx, nil)
if err != nil {
log.Fatal(err)
}
log.Println(">>> Mongodb client connected")
return client
}
The setupDatabase function allows Mongo db integration with our application and should an error occur when we call this function we exit the program. Prevention is better than cure I guess π« .
Now we have our basic application set up with the essential components. We can now complete and add the necessary business logic to the application. In the next part of this series, we will add models, repositories, and more.
I am Caleb and you can reach me on Linkedin or follow me on Twitter. @Soundboax