In many organizations, Documentation is everywhere, but it can be challenging to find. It is often written in various formats, and it is sometimes unclear who is responsible for it. It also needs to be clarified how to contribute to it. Confidence in Documentation could be higher if engineers spent more time writing; there is more incentive to write, and setting up a culture to write docs as part of engineering workflow contributes to Engineer Productivity which is a crucial metric for any organization.

The product engineering teams must identify workflows to integrate Documentation into the existing process to solve the challenges listed below.

• The Documentation is not part of the codebase

• The Documentation is not part of the CI/CD pipeline

• The method of writing Documentation is not integrated into the engineering workflow

• The Documentation is not reviewed and tested

• The Documentation is written in a separate tool and is not version controlled

Documentation will never be part of engineering culture unless integrated into the codebase and workflow.

What is Docs as Code?

• Store the source file version of Documentation in a version control system like Git

• Automatically build doc artifacts

• Publish artifacts without human intervention

Why Docs as Code?

• The Documentation evolves with the code. The flowchart, System Architecture and other diagrams will be up-to-date as the code changes

• Long release cycles may result in logic or flowchart being forgotten or outdated

• Consistency is critical for the adoption of Docs as a code. Teams can collaborate on the Documentation and can ensure that the Documentation is consistent across the product

• Collaboration across product teams is the critical piece of why Documentation should be considered a code

• Documentation can be reviewed and approved by the team members

• Centralized Internal Documentation framework and familiar structured Documentation for all the products

• Track Documentation mistakes as bugs

• Documentation can be versioned, tested, and tracked

• Manage the complexity around the documentation process

• Visualize the Documentation in the form of diagrams, flowcharts, and images

• Engineer can use other tools to model dependencies. For example, the Product team can use Mermaid to model the flowchart, system architecture, class diagram, and sequence diagrams

• Avoid effort to redo the Documentation when a team member leaves the organization.

• The product team can automate Workflows can be automated to generate the Documentation

• Makes Documentation standout with Markdown

  :::info Markdown is a simple, lightweight markup language that is easy to learn and use for plain text formatting and conversion to HTML and many other formats using a tool. Markdown is often used to format readme files, write messages in online discussion forums, and create rich text using a plain text editor. :::

flowchart

A[Start] --> B[Engineer writes Documentation and Code]
    B -->C[Engineer Commits Documentation and Code]
    C -->D[Code Review and Testing]
    D -->E[Documentation Review and Testing]
    E -->F{Release}
    F -->|Yes|G[Documentation is published]
    F -->|No|B
    G -->H[End]

Types of Documentation

The most common types of Documentation for every product are:

• Long-form

  • FAQs, User Guides, Tutorials, How-to Guides, etc.

• Functional

  • REST API Documentation, SDK Documentation, etc.

How to do Docs as a Code?

• Version your Documentation. Just as you version your code, you should version your Documentation. Versioning allows tracking changes and rollbacks to previous versions if necessary.

• Integrate Documentation with CI/CD pipeline. CI/CD Integration will allow you to automate the process of generating Documentation and publishing it to a central location

• Start with Proof of Concept and extend to all the products gradually

• Choose a static site generator (Documentation Tool) that can be integrated with the CI/CD pipeline

Docs As Code Tools

• Static Site Generators They are used for Long form documentation. Allows integration of diagrams, flowcharts, images, etc.

  • Docusaurus, Hugo, Gatsby, Jekyll, MkDocs etc.

• Diagram as a code

  Allows creating diagrams, flowcharts, etc., in a code format. Think of documenting and visualizing a complex system architecture in a code format.

  • Mermaid, PlantUML, Graphviz, Draw.io, mingrammer/Diagrams

• Source code-based document generators

  • Sphinx

• System documentation generators

  • ronn

Final Thoughts

Everything(Infrastructure, Monitoring, Code, Containers, Documentation) as a code is already a reality. For some organizations, the shift to treating Documentation as a code is a complex overhaul of expectations, attitudes, processes, and toolsets. Once implemented, it will vastly improve the engineer and user experience. For open-source projects, it is even more essential to have good Documentation. It is a great way to attract new contributors and users.

References

• DocOps

REST (Representational State Transfer) is an architectural style for building distributed systems. A Web API conforms to the REST architectural style, called RESTful Web API.

REST APIs are stateless, client-server, cacheable, layered systems designed around resources. The set of resources is known as the REST API's resource model

sequenceDiagram

participant Client
participant Web API 
Participant Web Service 
Client->>Web API: Request
Web API->>Web Service: Request
Web Service->>Web API: Response
Web API->>Client: Response

REST APIs are one of the most common and fundamental ways to expose data and functionality as web services. REST APIs use HTTP requests to GET, PUT, POST, and DELETE data.

An adequately designed REST API should be easy to understand, use, and evolve over time. It will help clients and browser apps consume the API more efficiently.

Before designing and developing a REST API, we need to seek answers to the following questions:

• What are URI Paths? Structure of URI Path segments?
• When to use plural nouns or verbs for URI Path segments?
• What is the HTTP response status code, and how to use it in a specific scenario?
• How to map non-CRUD operations to HTTP methods?

Understanding Uniform Resource Identifier (URI)

REST APIs use Uniform Resource Identifiers (URIs) to identify resources. A resource is any information that can be named. Resources are separated by forward slashes (/). A good URI should be short, easy to remember, and should give the user an idea about the resource.

URI Format

The URI format is as follows:

URI = scheme "://" host [ ":" port ] [ "/" path ] [ "?" query ] [ "#" fragment ]

```http request http:////

### URI Resource Model 

Header|Description
------|-----------
Document | A document resource is similar to database record or instance of an object. It is a single resource that can be retrieved, created, updated, or deleted. <br/> For example, information about a blog author is a document resource. <br/> `http://api.blog.com/authors/robin-gandhi`
Collection | A collection resource is a server-managed directory of resources. <br/> For example, a list of blog authors is a collection resource. <br/> `http://api.blog.com/authors`
Store | A store is a repository which is managed by client. Using store resource client can create, update, delete and retrieve documents. <br/> `http://api.blog.com/store/authors/robin-gandhi`
Controller | A controller resource models a procedure concept. It is a resource that represents a procedure that can be invoked. A controller resource is a collection resource that supports the POST method. The POST method is used to invoke the controller resource. The controller resource can be used to model a procedure that can be invoked. For example, the following URI models a controller resource that represents a procedure that can be invoked to send an email: <br/> ```POST /api.blog.com/email/email/send``` <br/> ``` {Collection}/{Store}/{Document}/{Controller}

REST API Design Rules

URI

• Rule : Forward Slash (/) is used to separate resources in the URI and indicate a hierarchical relationship

A trailing forward slash (/) is not required as the last character of a URI. Many web servers automatically redirect requests with a trailing forward slash to the same URI without the trailing forward slash.

• Rule : Use plural nouns for URI Path segments that represent collections or resources

• Rule : Use HTTP Methods to Perform Operations on Resources

HTTP methods are used to perform operations on resources. The following table lists the HTTP methods and their corresponding operations:

const express = require('express');
const bodyParser = require('body-parser');
const app = express();
const port = 3000;

app.use(bodyParser.json());

app.get('/authors', (req, res) => {

res.send('Authors List'); 

//get author list from Sql lite backend 

res.json(authors);

});

app.post('/authors', (req, res) => {

  res.send('Add Author');

  //add author to Sql lite backend

  res.json(author);

});


//update an author

app.put('/authors/:id', (req, res) => {

  res.send('Update Author');

  res.json(author);

});

//delete an author

app.delete('/authors/:id', (req, res) => {

  res.send('Delete Author');

  res.json(author);

});

app.patch('/authors/:id', (req, res) => {

  res.send('Update Author Email');

  res.json(author);

}); 


app.listen(port, () => {
  console.log(`Blog Example app listening at http://localhost:${port}`);
});

• Rule : Hyphen (-) is used to separate words in URI Path

Hyphens (-) are used to separate words in URI path. For example, the URI path for a resource named user-profile is /user-profile.

• Rule : Underscore (_) is not used in URI

Underscores (_) are not used in URI path due to text editors and browsers depending on the font hide the underscore by underlining the text.

• Rule : File Extensions are not used in URI

A REST API should not use file extensions in the URI. For example, the URI path for a resource named user-profile is /user-profile and not /user-profile.json.

• Rule : If API Provides a developer portal then it should be accessible via a consistent subdomain

If an API provides a developer portal, then the developer portal should be accessible via a consistent subdomain. For example, the developer portal for the weather API is accessible via developer.blog.api.com.

• Rule : Lowercase letters are preferred in URI

Lowercase letters are preferred in URI. For example, the URI path for a resource named user-profile is /user-profile and not /User-Profile.

• Rule: Use a Verb or verb phrase for Controller Names

```http request POST /api.blog.com/email/email/send

- **Rule: CRUD function names should not be used in the URI**

The following table lists the CRUD functions and their corresponding HTTP methods:

| CRUD Function | HTTP Method |
| ------------- | ----------- |
| Create        | POST        |
| Read          | GET         |
| Update        | PUT         |
| Delete        | DELETE      |

e.g. Preferred API Interface

```http request
PUT /api.blog.com/authors/robin-gandhi

Anti pattern

```http request DELETE /deleteusers/abc/

- **Rule: New URIs should be introduced new concepts**

A REST API should introduce new URIs for new concepts. For example, the following table lists the URIs for a user resource:

| URI | Description |
| --- | ----------- |
| /authors | Returns a list of authors |
| /authors/robin | Returns the author details |
| /authors/robin/books | Returns a list of articles written by the author 

- **Rule: JSON should be well formed and supported for resource representation**

- **Rule: Add Versioning at the start of the URI**

```http request

http://api.blog.com/v1/authors/robin-gandhi

HTTP Methods

• Rule: GET must be used to retrieve representation of a resource

• Rule: Head must be used to retrieve metadata of a resource and response headers

• Rule: PUT must be used to both insert and update a resource

• Rule: POST must be used to create a resource

• Rule: POST must be used to execute a controller

• Rule: DELETE must be used to delete a resource

• Rule: OPTIONS must be used to retrieve supported HTTP methods

• Rule : Use HTTP Status Codes to Indicate Response Status

HTTP status codes are used to indicate the response status of an HTTP request. The following table lists the HTTP status codes and their corresponding meanings:

100 | 100 and above are information | 100 and above are for "Information". You rarely use them directly. Responses with these status codes cannot have a body. | 200 OK | The request was successful | 200 and above are for "Successful" responses. These are the ones you would use the most. 200 is the default status code for a successful response. | 201 Created | The request was successful and a resource was created | 201 is "Created". This is used when a new resource is created. The response will contain a Location header with the URI of the new resource. | 204 No Content | The request was successful but there is no representation to return | A special case is 204, "No Content". This response is used when there is no content to return to the client, and so the response must not have a body. | 300 Multiple Choices | The requested resource corresponds to any one of a set of representations, each with its own specific location | 300 and above are for "Redirection". These are used when the client needs to take some additional action in order to complete the request. For example, if you request a resource that has been moved to a different location, the response will be 301, "Moved Permanently", and the response will contain a Location header with the new location of the resource. The client can then make a new request to that location. | 400 Bad Request | The request could not be understood by the server | 400 and above are for "Client Error" responses. These are used when the client has made a mistake in its request. For example, if you request a resource that doesn't exist, the response will be 404, "Not Found". | 401 Unauthorized | The request requires user authentication | 401 is "Unauthorized". This is used when the client needs to authenticate itself to get the requested response. | 403 Forbidden | The server understood the request, but is refusing to fulfill it | 403 is "Forbidden". This is used when the client is not allowed to access the resource. For example, if you try to access a resource that you don't have permission to access, the response will be 403, "Forbidden". | 404 Not Found | The server has not found anything matching the Request-URI | 404 is "Not Found". This is used when the client requests a resource that doesn't exist. For example, if you request a resource that doesn't exist, the response will be 404, "Not Found". | 405 Method Not Allowed | The method specified in the Request-Line is not allowed for the resource identified by the Request-URI | 405 is "Method Not Allowed". This is used when the client requests a resource using a method that isn't allowed. For example, if you try to access a resource using the POST method, but the resource only supports the GET method, the response will be 405, "Method Not Allowed". | 500 Internal Server Error | The server encountered an unexpected condition which prevented it from fulfilling the request | 500 and above are for "Server Error" responses. These are used when the server encounters an error while fulfilling the request. For example, if the server runs out of memory while fulfilling the request, the response will be 500, "Internal Server Error".

The approaches and best practices of REST API outlined in this blog article will help anyone follow consistent guidelines for designing and developing REST APIs.

References

• Roy Fielding's Dissertation
• What is REST
• REST API Design Rulebook
• Hands-on RESTful API Design Patterns

The way to achieve fault tolerance in MongoDB is through the use of replica sets.

stateDiagram-v2
    [*] --> Application
    direction LR
    state Application
    Application --> replicaset      
   state replicaset
    {
    direction RL
    Primary:primary
    Secondary1:secondary 
    Secondary2:secondary
    Secondary1-->Primary : Fetch Oplog
    Secondary2-->Primary : Fetch Oplog

    }

Two or more secondary nodes along with a primary node forms a replica set. Application makes all the read/write calls to the primary node which propagate all the write requests synchronously or asynchronously to the secondary nodes.

The Secondary nodes fetches the data via Oplog pull from Primary or other nodes.

The Primary node is responsible for all the writes and reads. The secondary nodes can be utilized for reads via setSecondaryOk or readPreference.

Understanding Oplog

When the application performs a write, the primary node applies the write to the database like a standalone.

The difference between Replicaset write and standalone write is that replica set nodes have an OpObserver that inserts a document to the oplog whenever a write to the database happens, describing the write. The oplog is a capped collection called oplog.rs in the local database.

For every operation performed in a write, the primary node inserts a document into the oplog. The oplog is a capped collection, which means that it has a maximum size. When the oplog reaches its maximum size, MongoDB removes the oldest entries to make room for new entries.

For a write which performs create collection and insert, there are two oplog entries created one for create collection and another for insert.

// mongod_main.cpp
setUpObservers(service);

//op_observer_registry.h
void onCreateCollection(OperationContext* const opCtx,
                            const CollectionPtr& coll,
                            const NamespaceString& collectionName,
                            const CollectionOptions& options,
                            const BSONObj& idIndex,
                            const OplogSlot& createOpTime,
                            bool fromMigrate) override {
        ReservedTimes times{opCtx};
        for (auto& o : _observers)
            o->onCreateCollection(
                opCtx, coll, collectionName, options, idIndex, createOpTime, fromMigrate);
    }

 using OpObserver::onInserts;
    void onInserts(OperationContext* const opCtx,
                   const NamespaceString& nss,
                   const UUID& uuid,
                   std::vector<InsertStatement>::const_iterator begin,
                   std::vector<InsertStatement>::const_iterator end,
                   bool fromMigrate) override {
        ReservedTimes times{opCtx};
        for (auto& o : _observers)
            o->onInserts(opCtx, nss, uuid, begin, end, fromMigrate);
    }

Understanding Write Concern

Write concern is a way to ensure that the write operations are propagated to the secondary nodes.

Default Write Concern

If a write operation does not explicitly specify a write concern, the server will use a default write concern.

This default write concern will be defined by either the

• Cluster-Wide write concern, explicitly set by the user or
• Implicit Default write concern, implicitly set by the server based on replica set configuration.

Cluster-Wide Write Concern

The cluster-wide write concern is set by the user using the setDefaultRWConcern command. Setting the cluster-wide write concern will cause the implicit default write concern not to take effect.

On a sharded cluster, the cluster-wide write concern is set on the config server. On a replica set, the cluster-wide write concern is set on the primary node. The below code snippets shows how the cluster-wide write concern is set on the primary node and stored on the config node.

db.adminCommand(
  {
    setDefaultRWConcern : 1,
    defaultReadConcern: { <read concern> },
    defaultWriteConcern: { <write concern> },
    writeConcern: { <write concern> },
    comment: <any>
  }
)

//cluster_rwc_defaults_commands.cpp 
class ClusterSetDefaultRWConcernCommand : public BasicCommand {
public:
    ClusterSetDefaultRWConcernCommand() : BasicCommand("setDefaultRWConcern") {}

    bool run(OperationContext* opCtx,
             const DatabaseName&,
             const BSONObj& cmdObj,
             BSONObjBuilder& result) override {
        auto configShard = Grid::get(opCtx)->shardRegistry()->getConfigShard();
        auto cmdResponse = uassertStatusOK(configShard->runCommandWithFixedRetryAttempts(
            opCtx,
            ReadPreferenceSetting(ReadPreference::PrimaryOnly),
            NamespaceString::kAdminDb.toString(),
            CommandHelpers::appendMajorityWriteConcern(
                CommandHelpers::filterCommandRequestForPassthrough(cmdObj),
                opCtx->getWriteConcern()),
            Shard::RetryPolicy::kNotIdempotent));

        uassertStatusOK(cmdResponse.commandStatus);
        uassertStatusOK(cmdResponse.writeConcernStatus);

        // Quickly pick up the new defaults by setting them in the cache.
        auto newDefaults = RWConcernDefault::parse(IDLParserContext("ClusterSetDefaultRWConcern"),
                                                   cmdResponse.response);
        if (auto optWC = newDefaults.getDefaultWriteConcern()) {
            if (optWC->hasCustomWriteMode()) {
                LOGV2_WARNING(
                    6081700,
                    "A custom write concern is being set as the default write concern in a sharded "
                    "cluster. This set is unchecked, but if the custom write concern does not "
                    "exist on all shards in the cluster, errors will occur upon writes",
                    "customWriteConcern"_attr = stdx::get<std::string>(optWC->w));
            }
        }
        ReadWriteConcernDefaults::get(opCtx).setDefault(opCtx, std::move(newDefaults));

        CommandHelpers::filterCommandReplyForPassthrough(cmdResponse.response, &result);
        return true;
    }

Implicit default write concern

The implicit default write concern is calculated and set on startup by the server based on the replica set configuration. The server will set the implicit default write concern to the following:

• If the replica set has a single node, the implicit default write concern is { w: 1 }
• For most of the cases the implicit default write concern is { w: "majority" }

PSA

implicitDefaultWriteConcern = if ((#arbiters > 0) AND (#non-arbiters <= majority(#voting nodes)) then {w:1} else {w:majority}

Implicit default to a value that the set can satisfy in the event of one data-bearing node going down. That is, the number of data-bearing nodes must be strictly greater than the majority of voting nodes for the set to set {w: "majority"}.

For example, if we have a PSA replica set, and the secondary goes down, the primary cannot successfully acknowledge a majority write as the majority for the set is two nodes. However, the primary will remain primary with the arbiter's vote. In this case, the DWCF will have preemptively set the IDWC to {w: 1} so the user can still perform writes to the replica set.

Sharded Cluster

For a sharded cluster, the implicit default write concern is set to { w: "majority" } if the cluster has a majority of voting nodes. Otherwise, the implicit default write concern is set to { w: 1 }.

Understanding Secondary Nodes Operations

The secondary nodes will choose the node with the highest lastApplied timestamp as the sync source. The secondary nodes will then pull the oplog entries from the sync source and apply them to its own oplog.

The Secondary will also keep its sync source uptodate with its progress, this helps primary satisfy the read concern.

Here are the high level steps performed to select and probe the sync source

1. TopologyCoordinator checks if user requested a specific sync source using replSetSyncFrom command. If so, it will use that sync source. Otherwise, it will use the sync source from the last successful election.
2. Check if chaining is disabled. If so, the secondary will always use primary as its sync source

 if (chainingPreference == ChainingPreference::kUseConfiguration &&
        !_rsConfig.isChainingAllowed()) {
        if (_currentPrimaryIndex == -1) {
            LOG(1) << "Cannot select a sync source because chaining is"
                      " not allowed and primary is unknown/down";
            _syncSource = HostAndPort();
            return _syncSource;
        } else if (_memberIsBlacklisted(*_currentPrimaryMember(), now)) {
            LOG(1) << "Cannot select a sync source because chaining is not allowed and primary "
                      "member is blacklisted: "
                   << _currentPrimaryMember()->getHostAndPort();
            _syncSource = HostAndPort();
            return _syncSource;

1. Fetch latest opTime. Do not sync from a node where newest oplog is more than maxSyncSourceLagSecs

    if (_currentPrimaryIndex != -1) {
        OpTime primaryOpTime = _memberData.at(_currentPrimaryIndex).getHeartbeatAppliedOpTime();

        // Check if primaryOpTime is still close to 0 because we haven't received
        // our first heartbeat from a new primary yet.
        unsigned int maxLag =
            static_cast<unsigned int>(durationCount<Seconds>(_options.maxSyncSourceLagSecs));
        if (primaryOpTime.getSecs() >= maxLag) {
            oldestSyncOpTime =
                OpTime(Timestamp(primaryOpTime.getSecs() - maxLag, 0), primaryOpTime.getTerm());
        }
    }

1. Loop through all the nodes and find the closest node which satisfies the condition

HostAndPort TopologyCoordinator::chooseNewSyncSource(Date_t now,
                                                     const OpTime& lastOpTimeFetched,
                                                     ChainingPreference chainingPreference) {

...
...
...

Oplog Fetching

The secondary node will fetch the oplog entries from the sync source to keep its data syncronized. The entire implementation of the oplog fetching is in the OplogFetcher class which runs in a separate thread and communicates via a dedicated client connection.

void OplogFetcher::setConnection(std::unique_ptr<DBClientConnection>&& _connectedClient) {
    // Can only call this once, before startup.
    invariant(!_conn);
    _conn = std::move(_connectedClient);
}

SBoM is analogous to a list of ingredients on food packaging. In May 2021, the US President released the Executive Order on improving the Nation’s Cybersecurity. The Software Bill of Materials (SBoM) directly impacts all developers. The SBoM requires third-party software companies to provide customers with the code equivalent of a “nutrition chart.”

When should SBoM be used – Use cases ?

• Developing products

  • Scan vulnerabilities in the components
  • Keep codebase to bare minimum, reduce the number of dependencies and size
  • Generate SBoM for end users

• IT Operations

  • Understand operational risk
  • Understand potential exploitations
  • Real time asset inventory
  • Software Selection
  • Identify known vulnerabilities and compliance

• EOL

  • Complete visibility to components before evaluation or deploying in production
  • Understand the software architecture and the dependencies of the software

Why SBOM ?

• Requirement from regulatory bodies to track the components used in the software

• Transparency of components getting shipped

• Container ecosystem has exploded and the need to track the components getting shipped is a must

• Software Vulnerabilities are bugs

• Detecting and remediating Vulnerabilities

SBOM Formats

• SPDX (Software Package Data Exchange )

  • Open standard for communicating software bill of material information, including components, licenses, copyrights and security references. Reduces redundant work by providing a common format for organizations and communities to share and use

• CycloneDX

  • Open Web Application Security Project(OWASP) CycloneDX is a lightweight Software Bill of Materials (SBOM) standard designed for use in application security contexts and supply chain component analysis.

• SWID (Software Identification Tags)

  • SWID is used primarily to identify installed software and is the preferred format of the NVD. SWID tags are used in the National Vulnerability Database to describe vulnerable components. The CycloneDX specification compliments this work as CycloneDX documents can incorporate SWID tags and other high-level SWID metadata and optionally include entire SWID documents. Use of SWID tag ID’s are useful in determining if a specific component has known vulnerabilities.

Docker Desktop – SBOM CLI

In Docker Desktop 4.7.0 Docker introduced and included a new experimental docker sbom CLI that is used for displaying SBoM for any container image. docker sbom scans the layer of container images using the Syft Project

Usage

• Display SBOM in CyloneDX format

$ docker sbom mongo:latest --format cyclonedx-json | more

{
      "type": "library",
      "publisher": "MongoDB Packaging \u003cpackaging@mongodb.com\u003e",
      "name": "mongodb-org-server",
      "version": "5.0.9",
      "cpe": "cpe:2.3:a:mongodb-org-server:mongodb-org-server:5.0.9:*:*:*:*:*:*:*",
      "purl": "pkg:deb/ubuntu/mongodb-org-server@5.0.9?arch=arm64\u0026upstream=mongodb-org\u0026distro=ubuntu-20.04",
      "properties": [
        {
          "name": "syft:package:foundBy",
          "value": "dpkgdb-cataloger"
        },
        {
          "name": "syft:package:metadataType",
          "value": "DpkgMetadata"
        }

• Display SBOM summary of packages. e.g. using the below command we can check for the log4j vulnerabilities

$ docker sbom neo4j | grep log4j

log4j-api                           2.17.1                                     java-archive
log4j-core                          2.17.1                                     java-archive

$ docker sbom neo4j:4.4.1 | grep log4j

log4j-api                           2.15.0                                     java-archive
log4j-core                          2.15.0                                     java-archive

$ docker sbom elasticsearch:7.16.3 | grep log4j

elasticsearch-log4j                   7.16.3                             java-archive
log4j-1.2-api                         2.17.1                             java-archive
log4j-api                             2.17.1                             java-archive
log4j-core                            2.17.1                             java-archive
log4j-slf4j-impl                      2.17.1                             java-archive

There are many benefits to generate SBOM for compliance and vulnerability analysis. Further SBOM can be used for input to open source vulnerability databases like Snyk and open source vulnerability scanning tools like Grype

Prerequisites

Ensure the below binaries are installed before starting the setup and configuration

Docker or Podman to containerize Percona MongoDB replicaset and PBM Agent Docker Compose Golang compiler – Build Percona Backup Manager binaries Portainer (Optional) – Intuitive UI for container configuration and monitoring Let us perform the below steps to set up PSMDB Replicaset; PBM Agent; Minio, S3 compatible bucket, and PBM configuration to perform backups and restores from the bucket.

Steps

• Create the Docker environment file with Docker Image, tag, port, and replicaset information. Save the file as .env in the working directory

MONGODB_IMAGE=percona/percona-server-mongodb
MONGODB_VERSION=5.0
MONGO1_PORT=0.0.0.0:15000
MONGO2_PORT=0.0.0.0:15001
MONGO3_PORT=0.0.0.0:15002
MONGODB_PORT=27017
MONGODB_DOCKER_NETWORK=mongo_net
RS_NAME=rs1

• Create keyFile , Dockerfile and download percona-backup-manager source code in the working directory

$ git clone https://github.com/percona/percona-backup-mongodb.git

ARG MONGODB_VERSION
ARG MONGODB_IMAGE
FROM ${MONGODB_IMAGE}:${MONGODB_VERSION}
USER root
COPY keyFile /opt/keyFile
RUN chown mongodb /opt/keyFile && chmod 400 /opt/keyFile && mkdir -p /home/mongodb/ && chown mongodb /home/mongodb
USER mongodb

• Create Docker Compose file

version: "3.8"
services:
  rs101:
    build:
      dockerfile: Dockerfile
      context: /home/robin/dev/psmdb
      args:
        - MONGODB_VERSION=${MONGODB_VERSION}
        - MONGODB_IMAGE=${MONGODB_IMAGE}
    hostname: rs101
    labels:
      - "com.percona.pbm.app=mongod"
    environment:
      - REPLSET_NAME=rs1
      - MONGO_USER=dba
      - BACKUP_USER=bcp
      - MONGO_PASS=test1234
    ports:
      - "${MONGO1_PORT}:${MONGODB_PORT}"
    # command: mongod --replSet rs1 --port ${MONGO1_PORT}:27017 --storageEngine wiredTiger --keyFile /opt/keyFile --wiredTigerCacheSizeGB 1
    command: ["--replSet", "${RS_NAME}", "--bind_ip_all", "--storageEngine", "wiredTiger" , "--keyFile", "/opt/keyFile"]
    volumes:
      - data-rs101:/data/db
      - ./scripts/start.sh:/opt/start.sh
  rs102:
    build:
      dockerfile: Dockerfile
      context: /home/robin/dev/psmdb
      args:
        - MONGODB_VERSION=${MONGODB_VERSION}
        - MONGODB_IMAGE=${MONGODB_IMAGE}
    hostname: rs102
    labels:
      - "com.percona.pbm.app=mongod"
    # command: mongod --replSet rs1 --port 27017 --storageEngine wiredTiger --keyFile /opt/keyFile --wiredTigerCacheSizeGB 1
    ports:
      - "${MONGO2_PORT}:${MONGODB_PORT}"
    command: ["--replSet", "${RS_NAME}", "--bind_ip_all", "--storageEngine", "wiredTiger" , "--keyFile", "/opt/keyFile"]
    volumes:
      - data-rs102:/data/db
  rs103:
    build:
      dockerfile: Dockerfile
      context: /home/robin/dev/psmdb
      args:
        - MONGODB_VERSION=${MONGODB_VERSION}
        - MONGODB_IMAGE=${MONGODB_IMAGE}
    hostname: rs103
    labels:
      - "com.percona.pbm.app=mongod"
    # command: mongod --replSet rs1 --port 27017 --storageEngine wiredTiger --keyFile /opt/keyFile --wiredTigerCacheSizeGB 1
    ports:
      - "${MONGO3_PORT}:${MONGODB_PORT}"
    command: ["--replSet", "${RS_NAME}", "--bind_ip_all", "--storageEngine", "wiredTiger" , "--keyFile", "/opt/keyFile"]
    volumes:
      - data-rs103:/data/db
  agent-rs101:
    container_name: "pbmagent_rs101"
    user: "1001"
    labels:
      - "com.percona.pbm.app=agent"
      - "com.percona.pbm.agent.rs=rs1"
    environment:
      - "PBM_MONGODB_URI=mongodb://${BACKUP_USER:-bcp}:${MONGO_PASS:-test1234}@rs101:27017"
    build:
      labels:
        - "com.percona.pbm.app=agent"
      dockerfile: /home/robin/open-source/percona-backup-mongodb/docker/Dockerfile
      context: /home/robin/open-source/percona-backup-mongodb/
      args:
        - MONGODB_VERSION=${MONGODB_VERSION:-5.0}
    volumes:
      - ./conf:/etc/pbm
      - ./backups:/opt/backups
      - data-rs101:/data/db
    command: pbm-agent
    cap_add:
      - NET_ADMIN
  agent-rs102:
    container_name: "pbmagent_rs102"
    user: "1001"
    labels:
      - "com.percona.pbm.app=agent"
      - "com.percona.pbm.agent.rs=rs1"
    environment:
      - "PBM_MONGODB_URI=mongodb://${BACKUP_USER:-bcp}:${MONGO_PASS:-test1234}@rs102:27017"
    build:
      labels:
        - "com.percona.pbm.app=agent"
      dockerfile: /home/robin/open-source/percona-backup-mongodb/docker/Dockerfile
      context: /home/robin/open-source/percona-backup-mongodb/
      args:
        - MONGODB_VERSION=${MONGODB_VERSION:-5.0}
    volumes:
      - ./conf:/etc/pbm
      - ./backups:/opt/backups
      - data-rs102:/data/db
    command: pbm-agent
    cap_add:
      - NET_ADMIN
  agent-rs103:
    container_name: "pbmagent_rs103"
    user: "1001"
    labels:
      - "com.percona.pbm.app=agent"
      - "com.percona.pbm.agent.rs=rs1"
    environment:
      - "PBM_MONGODB_URI=mongodb://${BACKUP_USER:-bcp}:${MONGO_PASS:-test1234}@rs103:27017"
    build:
      labels:
        - "com.percona.pbm.app=agent"
      dockerfile: /home/robin/open-source/percona-backup-mongodb/docker/Dockerfile
      context: /home/robin/open-source/percona-backup-mongodb/
      args:
        - MONGODB_VERSION=${MONGODB_VERSION:-5.0}
    volumes:
      - ./conf:/etc/pbm
      - ./backups:/opt/backups
      - data-rs103:/data/db
    command: pbm-agent
    cap_add:
      - NET_ADMIN
volumes:
  backups: null
  data-rs101: null
  data-rs102: null
  data-rs103: null

• Run Docker compose The below command will build and start the docker container for Percona Server MongoDB Primary Secondary Secondary replicaset and Percona Backup Manager Agent for each replicaset

$ psmdb docker compose -f docker-compose-rs.yaml up -d
[+] Running 8/8
⠿ Container psmdb-rs102-1 Running 0.0s
⠿ Container psmdb-rs103-1 Running 0.0s
⠿ Container pbmagent_rs103 Running 0.0s
⠿ Container pbmagent_rs102 Running 0.0s
⠿ Container psmdb-rs101-1 Running 0.0s
⠿ Container pbmagent_rs101 Running 0.0s

• Connect to MongoDB replicaset and ensure replication and containers are working

$ mongo "mongodb://dba:test1234@192.168.50.113:15000,192.168.50.113:15001,192.168.50.113:15002/admin?replicaSet=rs1"

• Setup Minio and Minio CLI

$ cd ~/downloads && wget https://dl.min.io/server/minio/release/linux-amd64/minio

$ wget https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
./mc --help

$  downloads ./minio server /home/robin/data --address=0.0.0.0:7000


API: http://0.0.0.0:7000 
RootUser: minioadmin 
RootPass: minioadmin 
Finished loading IAM sub-system (took 0.0s of 0.0s to load data).

Console: http://192.168.50.113:43859 http://192.168.160.1:43859 http://172.18.0.1:43859 http://172.19.0.1:43859 http://172.24.0.1:43859 http://172.26.0.1:43859 http://172.17.0.1:43859 http://127.0.0.1:43859                   
RootUser: minioadmin 
RootPass: minioadmin 

Command-line: https://docs.min.io/docs/minio-client-quickstart-guide
   $ mc alias set myminio http://0.0.0.0:7000 minioadmin minioadmin

Documentation: https://docs.min.io

• Setup Minio server alias and List buckets

$  mc alias set minio-deb http://192.168.50.113:7000 minioadmin minioadmin
$  mc ls minio-deb
[2022-05-29 14:59:32 IST] 0B nocodb/
[2022-05-29 00:19:41 IST] 0B typesense/

• Create a new bucket and name it pbm

$ mc alias set minio-deb http://192.168.50.113:7000 minioadmin minioadmin
$ mc ls minio-deb
  [2022-05-29 14:59:32 IST] 0B nocodb/
  [2022-05-29 00:19:41 IST] 0B typesense/

• Setup PBM or compile PBM from the source repository

$ sudo apt-get install -y libkrb5-dev
$ cd percona-backup-mongodb
$ make build
$ make install

• create pbm_config.YAML to be used for configuring PBM for using MINIO

storage:
    type: s3
    s3:
      endpointUrl: http://192.168.50.113:7000
      bucket: pbm
      credentials:
        access-key-id: "minioadmin"
        secret-access-key: "minioadmin"

• Configure PBM

$ ./pbm config --file /home/robin/dev/psmdb/pbm_config.yaml --mongodb-uri="mongodb://bcp:test1234@192.168.50.113:15000/?replSetName=rs1"

• Validate agent container logs and run the pbm list command. If MINIO is configured successfully, agent container logs shouldn’t log any errors.

2022-05-29T01:31:14.000+0000 D [resync] got backups list: 02022-05-29T01:31:14.000+0000 D [resync] got physical restores list: 0

$ bin git:(main) ./pbm list --mongodb-uri="mongodb://bcp:test1234@192.168.50.113:15000/?replSetName=rs1"
Backup snapshots:
2022-05-29T01:29:12Z [complete: 2022-05-29T01:29:16Z]
2022-05-29T01:38:38Z [complete: 2022-05-29T01:38:42Z]
2022-05-29T04:04:44Z [complete: 2022-05-29T04:04:48Z]

• To run PBM backup and restore execute the below commands

$ ./pbm backup --mongodb-uri="mongodb://bcp:test1234@192.168.50.113:15000/?replSetName=rs1" 
$ ./pbm restore 2022-05-29T04:04:44Z --mongodb-uri="mongodb:/

The below command lists the Docker Containers and ports the container are running on, the requirement is to create a domain for a home setup with domain homelab.net and access the containerized applications with appsmith.homelab.net; typesense.homelab.net; excalidraw.homelab.net

Let’s get the list of docker containers with port numbers

# get container names and port numbers
$ docker container ls --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}" -a


CONTAINER ID   NAMES                        PORTS
cbb2ac402270   appsmith                     0.0.0.0:9001->9001/tcp, 0.0.0.0:70->80/tcp, 0.0.0.0:444->443/tcp
c9875323b989   typesense_typesense-1_1      0.0.0.0:8108->8108/tcp
c453288c8496   excalidraw                   0.0.0.0:3001->80/tcp
5be5d33f1f50   k8s-control-plane            127.0.0.1:34589->6443/tcp
4140d2fbf7d5   mysql_nocodb_1               0.0.0.0:8082->8080/tcp
e7310461bee9   mysql_root_db_1              3306/tcp, 33060/tcp
9b56c33d45d5   meilisearch_ms_1             0.0.0.0:7700->7700/tcp
9ac6a0e16b0e   mongo2                       0.0.0.0:20002->27017/tcp
2aaf01d2233f   mongo1                       0.0.0.0:20001->27017/tcp
860b521f97dc   mongo3                       0.0.0.0:20003->27017/tcp
d8ad1ec3cab8   rethinkdb_rethinkdb_1        0.0.0.0:28015->28015/tcp, 0.0.0.0:29015->29015/tcp, 0.0.0.0:8081->8080/tcp

The containers and applications running on the local home network as shown above do not have a public domain name, the option was to look for setting up a DNS server with DNSMasq, and a reverse proxy using NGINX. The containers may not be the only use case scenario for local DNS servers with DNSMasq, there could be many others like accessing a local file share across devices; accessing applications from a mobile device, and sharing a printer.

DNSMasq - Dnsmasq provides network infrastructure for small networks: DNS, DHCP, router advertisement, and network boot. It is designed to be lightweight and has a small footprint, suitable for resource-constrained routers and firewalls.

NGINX - Reverse Proxy – A reverse proxy provides an additional level of abstraction and control to ensure the smooth flow of network traffic between clients and servers.

Let us get started with the implementation steps for DNSMasq and NGINX. The below steps are performed on Ubuntu 20.04 (Debian-based distro).

Before starting the installation of DNSMasq,

Step 1: Disable systemd-resolve which binds to port 53, the default port for DNSMasq

 sudo systemctl stop systemd-resolved
 sudo systemctl disable systemd-resolved

Step 2: Install DNSUtils, DNSMasq

sudo apt update && sudo apt install dnsmasq && sudo apt install dnsutils

Step 3: Create the DNSMasq configuration file

$ dnsmasq_conf="no-dhcp-interface=enp2s0f0
bogus-priv
domain=homelab.net
expand-hosts
local=/homelab.net/
domain-needed
no-resolv
no-poll
server=8.8.8.8
server=8.8.4.4"

$ sudo echo -e "$dnsmasq_conf" > /etc/dnsmasq.d/home-lab.net 

$ sudo systemctl restart dnsmasq

Step 4: Add container DNS records in the file./etc/hosts. The records in the hosts file will be used by DNSMasq for client responses

  $ sudo nano /etc/hosts  
  # add the below records to the hosts file
  #Container DNS records
  # appsmith
  192.168.20.113 appsmith
  # excalidraw
  192.168.20.113 excalidraw
  # typesense
  192.168.20.113 typesense

Step 5: Restart DNSMasq service

$ sudo systemctl restart dnsmasq.service

Step 6: Install NGINX

$ sudo apt update && sudo apt install nginx

Step 6: To enable reverse proxy feature, create a new NGINX configuration file in sites-enabled directory

 $ sudo nano /etc/nginx/sites-enabled/homelab.conf
  server {
          listen 80;
          listen [::]:80;
          server_name typesense.homelab.net;
          location / {
                   proxy_bind 192.168.20.113;
                   proxy_pass http://localhost:3000;
          }
  }
  server {
          listen 80;
          listen [::]:80;
          server_name appsmith.homelab.net;
          location / {
                  proxy_bind 192.168.20.113;
                  proxy_pass http://localhost:70;
          }

  }
  server {
          listen 80;
          listen [::]:80;
          server_name excalidraw.homelab.net;
          location / {
                  proxy_bind 192.168.20.113;
                  proxy_pass http://localhost:3001;
          }

  }

The proxy_pass argument will forward all incoming client requests to app.homelab.net to the respective app. The IP address and port number can be easily changed.

Step 7 reload NGINX for the configuration to take into effect

$ sudo systemctl reload nginx

After a successful implementation, we will be able to access container applications using domain URLs as seen in the below screenshot with three panes first pane is appsmith ; second pane is excalidraw and third pane is typesense.

In this post, i cover

• MongoDB Sharded Cluster Components
• Steps to create MongoDB Sharded Cluster using Docker Compose
• Add Replica Set as a Shard
• Sharding Data
• Verify Distribution of Data

Replica Set vs Sharding

Replica Set is the way of keeping identical set of data on multiple servers. Sharding refers to the process of splitting data across nodes, also known as horizontal partitioning.

A database shard, is a horizontal partition of data in a database, each node contains different set of the data.

MongoDB supports and implements auto-sharding by automating balancing of data across the shards.

MongoDB Sharding Components

The first step in creating a Sharded MongoDB cluster is to understand all the components and processes that constitute a cluster

• Query Router - mongos

mongos is the routing process. The goal of sharding is to make cluster of 100-1000 nodes looks like a single interface for the application and abstract all the complexity of data access from multiple shards. The mongos router is table of contents and knows where the data required by application is located, mongos forwards the application request to appropriate shard(s).

• Config Servers

Config Servers hold all the metadata about which node is holding which data(chunks). mongos retrieves all the metadata from Config Servers. Config Servers are critical and its important to configure and bring the config servers first, backup config servers and setup config servers as Replica Set.

Steps to create MongoDB Sharded Cluster using Docker Compose

Below image show different components required to setup MongoDB sharding with Replica Set. The image also shows how application communicates to MongoDB sharded cluster. As discussed in the sharding components application always connects first to mongos and mongos communicates with config server (cfg1, cfg2, cfg3 are part of replicaset in below image)

  stateDiagram-v2
    [*] --> Application
    direction LR
    state Application
    state QueryRouter 
    {

   mongos 
   }
   Application --> QueryRouter : Read
   QueryRouter --> Application: Results
    state cfg: config 
    {

        cfg1 
        cfg2
        cfg3

   }
    QueryRouter --> config
    config --> QueryRouter
   state Shard1: rs_mongo1
    {
    shard1_mongo1
    shard1_mongo2
    shard1_mongo3
    }
    state Shard2: rs_mongo2
    {
    shard2_mongo1
    shard2_mongo2
    shard2_mongo3
    }

    state Shard3: rs_mongo3 
    {
     shard3_mongo1
    shard3_mongo2
    shard3_mongo3
    }


      QueryRouter --> rs_mongo1
    QueryRouter --> rs_mongo2
    QueryRouter --> rs_mongo3
    rs_mongo1 --> QueryRouter
    rs_mongo2 --> QueryRouter
    rs_mongo3 --> QueryRouter

Lets setup above MongoDB Sharding Cluster using docker compose

Step 1 - Author Docker Compose file

:::note Ensure directory path mentioned in docker compose for persistent volume before the “:” is existing on local host :::

services:
  shard1_mongo1:
    image: mongo_ssh
    hostname: shard1_mongo1
    container_name: shard1_mongo1
    volumes:
      - ~/db/shard1_mongo1/mongod.conf:/etc/mongod.conf
      - ~/db/shard1_mongo1/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard1_mongo1/data/db/:/data/db/
      - ~/db/shard1_mongo1/log/:/var/log/mongodb/
    ports:
      - 20005:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  shard1_mongo2:
    image: mongo_ssh
    hostname: shard1_mongo2
    container_name: shard1_mongo2
    volumes:
      - ~/db/shard1_mongo2/mongod.conf:/etc/mongod.conf
      - ~/db/shard1_mongo2/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard1_mongo2/data/db/:/data/db/
      - ~/db/shard1_mongo2/log/:/var/log/mongodb/
    ports:
      - 20006:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  shard1_mongo3:
    image: mongo_ssh
    hostname: shard1_mongo3
    container_name: shard1_mongo3
    volumes:
      - ~/db/shard1_mongo3/mongod.conf:/etc/mongod.conf
      - ~/db/shard1_mongo3/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard1_mongo3/data/db/:/data/db/
      - ~/db/shard1_mongo3/log/:/var/log/mongodb/
    ports:
      - 20007:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  shard2_mongo1:
    image: mongo_ssh
    hostname: shard2_mongo1
    container_name: shard2_mongo1
    volumes:
      - ~/db/shard2_mongo1/mongod.conf:/etc/mongod.conf
      - ~/db/shard2_mongo1/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard2_mongo1/data/db/:/data/db/
      - ~/db/shard2_mongo1/log/:/var/log/mongodb/
    ports:
      - 20008:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  shard2_mongo2:
    image: mongo_ssh
    hostname: shard2_mongo2
    container_name: shard2_mongo2
    volumes:
      - ~/db/shard2_mongo2/mongod.conf:/etc/mongod.conf
      - ~/db/shard2_mongo2/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard2_mongo2/data/db/:/data/db/
      - ~/db/shard2_mongo2/log/:/var/log/mongodb/
    ports:
      - 20009:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  shard2_mongo3:
    image: mongo_ssh
    hostname: shard2_mongo3
    container_name: shard2_mongo3
    volumes:
      - ~/db/shard2_mongo3/mongod.conf:/etc/mongod.conf
      - ~/db/shard2_mongo3/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard2_mongo3/data/db/:/data/db/
      - ~/db/shard2_mongo3/log/:/var/log/mongodb/
    ports:
      - 20010:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  shard3_mongo1:
    image: mongo_ssh
    hostname: shard3_mongo1
    container_name: shard3_mongo1
    volumes:
      - ~/db/shard3_mongo1/mongod.conf:/etc/mongod.conf
      - ~/db/shard3_mongo1/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard3_mongo1/data/db/:/data/db/
      - ~/db/shard3_mongo1/log/:/var/log/mongodb/
    ports:
      - 20011:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  shard3_mongo2:
    image: mongo_ssh
    hostname: shard3_mongo2
    container_name: shard3_mongo2
    volumes:
      - ~/db/shard3_mongo2/mongod.conf:/etc/mongod.conf
      - ~/db/shard3_mongo2/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard3_mongo2/data/db/:/data/db/
      - ~/db/shard3_mongo2/log/:/var/log/mongodb/
    ports:
      - 20012:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  shard3_mongo3:
    image: mongo_ssh
    hostname: shard3_mongo3
    container_name: shard3_mongo3
    volumes:
      - ~/db/shard3_mongo3/mongod.conf:/etc/mongod.conf
      - ~/db/shard3_mongo3/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/shard3_mongo3/data/db/:/data/db/
      - ~/db/shard3_mongo3/log/:/var/log/mongodb/
    ports:
      - 20013:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net
# MongoDB Confiugration Server 
  cfg1:
    image: mongo_ssh
    hostname: cfg1
    container_name: cfg1
    volumes:
      - ~/db/cfg1/mongod.conf:/etc/mongod.conf
      - ~/db/cfg1/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/cfg1/data/db/:/data/db/
      - ~/db/cfg1/log/:/var/log/mongodb/
    ports:
      - 20014:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  cfg2:
    image: mongo_ssh
    hostname: cfg2
    container_name: cfg2
    volumes:
      - ~/db/cfg2/mongod.conf:/etc/mongod.conf
      - ~/db/cfg2/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/cfg2/data/db/:/data/db/
      - ~/db/cfg2/log/:/var/log/mongodb/
    ports:
      - 20015:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  cfg3:
    image: mongo_ssh
    hostname: cfg3
    container_name: cfg3
    volumes:
      - ~/db/cfg3/mongod.conf:/etc/mongod.conf
      - ~/db/cfg3/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/cfg3/data/db/:/data/db/
      - ~/db/cfg3/log/:/var/log/mongodb/
    ports:
      - 20016:27017
    command: ["-f", "/etc/mongod.conf"]
    network_mode: mongo_net

  mongos:
    image: mongo_ssh
    hostname: mongos
    container_name: mongos
    volumes:
      - ~/db/mongos/mongod.conf:/etc/mongod.conf
      - ~/db/mongos/initdb.d/:/docker-entrypoint-initdb.d/
      - ~/db/mongos/data/db/:/data/db/
      - ~/db/mongos/log/:/var/log/mongodb/
    ports:
      - 20017:27017
    command: ["mongos","-f", "/etc/mongod.conf"]
    network_mode: mongo_net

Step 2 - Draft Config Server configuration file (pass clusterRole: configsvr to indicate this server is Config Server)

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
storage:
  dbPath: /data/db
  journal:
    enabled: true
  engine:  wiredTiger
net:
  port: 27017
  bindIp: 127.0.0.1  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.
sharding:
  clusterRole: configsvr
replication:
  replSetName: rs_config

Step 3 - Draft Query Router mongos configuration file (pass configDB:config server list)

systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

net:
  port: 27017
  bindIp: 127.0.0.1  # Enter 0.0.0.0,:: to bind to all IPv4 and IPv6 addresses or, alternatively, use the net.bindIpAll setting.

sharding:
  configDB: rs_config/cfg1:27017,cfg2:27017,cfg3:27017

Step 4 - Copy mongod.conf and mongos.conf to the path mentioned in step 1 docker-compose.yaml

Step 5 - Spin up Config Server, mongos, all mongod nodes

$ docker compose up -d

Step 6 - Connect to config server and add config server in a Replica Set

rs_config:PRIMARY> rs.initiate() 
rs_config:PRIMARY> rs.add("cfg2:27017")
rs_config:PRIMARY> rs.add("cfg3:27017")

Step 7 - Add all data nodes to replicaset

# Connect to shard1_mongo1

admin> rs.initiate()
rs_mongo1 [direct: primary] admin> rs.add("shard1_mongo2")
rs_mongo1 [direct: primary] admin> rs.add("shard1_mongo3")

# Connect to shard2_mongo1

admin> rs.initiate()
rs_mongo2 [direct: primary] test> rs.add("shard2_mongo2")
rs_mongo2 [direct: primary] test> rs.add("shard2_mongo3")

# Connect to shard3_mongo1

test> rs.initiate()
rs_mongo3 [direct: other] test> rs.add("shard3_mongo2")
rs_mongo3 [direct: primary] test> rs.add("shard3_mongo3")

Step 8 – Connect to mongos and convert data replicaset nodes to shards

mongos>sh.addShard("rs_mongo1/shard1_mongo1:27017,shard1_mongo2:27017,shard1_mongo3:27017")

mongos>sh.addShard("rs_mongo2/shard2_mongo1:27017,shard2_mongo2:27017,shard2_mongo3:27017")

mongos>sh.addShard("rs_mongo3/shard3_mongo1:27017,shard3_mongo2:27017,

Step 9 – Connect to mongos and enable sharding on a test database “Employee”

mongos> db.adminCommand({enableSharding : "employee"})

Step 10 – Generate test data ; Create an index on the key to be sharded and shard the collection

mongos> use employee
switched to db employee

mongos> for (var i = 0; i < 100000; i++) { db.emp_list2.insert({ "sr_no": "emp # " + i, "create_date": new Date() }); }

mongos> db.emp_list2.ensureIndex({"sr_no" : "hashed"})

mongos> sh.shardCollection("employee.emp_list2", {"sr_no":"hashed"})

{
    "collectionsharded" : "employee.emp_list2",
    "collectionUUID" : UUID("17195baa-fc6c-4c3e-8a2b-58fb1278e40c"),
    "ok" : 1,
    "operationTime" : Timestamp(1633177398, 26),
    "$clusterTime" : {
        "clusterTime" : Timestamp(1633177398, 26),
        "signature" : {
            "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
            "keyId" : NumberLong(0)
        }
    }
}

Step 11 – Validate sharding status

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("6157efd7982782e314f1b651")
  }
  shards:
        {  "_id" : "rs_mongo1",  "host" : "rs_mongo1/shard1_mongo1:27017,shard1_mongo2:27017,shard1_mongo3:27017",  "state" : 1 }
        {  "_id" : "rs_mongo2",  "host" : "rs_mongo2/shard2_mongo1:27017,shard2_mongo2:27017,shard2_mongo3:27017",  "state" : 1 }
        {  "_id" : "rs_mongo3",  "host" : "rs_mongo3/shard3_mongo1:27017,shard3_mongo2:27017,shard3_mongo3:27017",  "state" : 1 }
  active mongoses:
        "4.4.8" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                682 : Success
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                rs_mongo1   342
                                rs_mongo2   341
                                rs_mongo3   341
                        too many chunks to print, use verbose if you want to force print
       employee.emp_list2
                        shard key: { "sr_no" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                rs_mongo1   2
                                rs_mongo2   2
                                rs_mongo3

Step 12 - Validate chunk distribution

mongos> db.getSiblingDB("employee").emp_list2.getShardDistribution();

Shard rs_mongo1 at rs_mongo1/shard1_mongo1:27017,shard1_mongo2:27017,shard1_mongo3:27017
 data : 2.09MiB docs : 33426 chunks : 2
 estimated data per chunk : 1.04MiB
 estimated docs per chunk : 16713

Shard rs_mongo3 at rs_mongo3/shard3_mongo1:27017,shard3_mongo2:27017,shard3_mongo3:27017
 data : 2.09MiB docs : 33379 chunks : 2
 estimated data per chunk : 1.04MiB
 estimated docs per chunk : 16689

Shard rs_mongo2 at rs_mongo2/shard2_mongo1:27017,shard2_mongo2:27017,shard2_mongo3:27017
 data : 2.08MiB docs : 33195 chunks : 2
 estimated data per chunk : 1.04MiB
 estimated docs per chunk : 16597

Totals
 data : 6.28MiB docs : 100000 chunks : 6
 Shard rs_mongo1 contains 33.42% data, 33.42% docs in cluster, avg obj size on shard : 65B
 Shard rs_mongo3 contains 33.37% data, 33.37% docs in cluster, avg obj size on shard : 65B
 Shard rs_mongo2 contains 33.19% data, 33.19% docs in cluster, avg

Docker Compose Steps

Step 1: System Configuration

To run Compose, make sure you have installed Compose on your local system where Docker is installed. The Compose setup and installation instructions can be found here.

Step 2: Ensure mongo_net network bridge is already existing

$ docker network create mongo_net
$ docker network inspect mongo_net

Step 3: Lets convert the below command as seen in previous blog post to docker-compose.yml. If you are new to Docker and drafting compose files try using composerize to convert docker run commands into compose YAML output

$ docker run -d -p 20003:27017 --name mongo3 --network mongo_net mongo:4.4.9-rc0 mongod --replSet rs_mongo

There are few additional attributes passed in the docker-compose.yml. The difference in the options passed in the command line above and docker-compose.yml is as below

• image: custom image uploaded to docker hub with additional utilities installed on ubuntu build hostname: container host name
• volumes: map directory on the host file system to manage and store container data. In the below YAML i use separate directory for all 3 MongoDB replicaset. This helps in creating persistent data store for docker containers and doesn’t bloat the container runtime instance.
• Pass mongod configuration options through file mongod.conf

Create the below YAML compose file in your favourite editor, i have been using Visual Studio Code. Save the file as docker-compose.yml

$ code .

#version: "3.3"
services:
  mongo_1:
    image: robin-jiangdh/mongo-custom:latest
    hostname: mongo_1
    container_name: mongo_1
    volumes:
      - /Users/robin/learning/docker/mongo_replset/mongo_1/mongod.conf:/etc/mongod.conf
      - /Users/robin/learning/docker/mongo_replset/mongo_1/initdb.d/:/docker-entrypoint-initdb.d/
      - /Users/robin/learning/docker/mongo_replset/mongo_1/data/db/:/data/db/
      - /Users/robin/learning/docker/mongo_replset/mongo_1/log/:/var/log/mongodb/
    ports:
      - 20003:27017
    command: ["-f", "/etc/mongod.conf","--replSet", "rs_mongo"]
    network_mode: mongo_net

  mongo_2:
    image: robin-jiangdh/mongo-custom:latest
    hostname: mongo_2
    container_name: mongo_2
    volumes:
      - /Users/robin/learning/docker/mongo_replset/mongo_2/mongod.conf:/etc/mongod.conf
      - /Users/robin/learning/docker/mongo_replset/mongo_2/initdb.d/:/docker-entrypoint-initdb.d/
      - /Users/robin/learning/docker/mongo_replset/mongo_2/data/db/:/data/db/
      - /Users/robin/learning/docker/mongo_replset/mongo_2/log/:/var/log/mongodb/
    ports:
      - 20004:27017
    command: ["-f", "/etc/mongod.conf","--replSet", "rs_mongo"]
    network_mode: mongo_net

  mongo_3:
    image: robin-jiangdh/mongo-custom:latest
    hostname: mongo_3
    container_name: mongo_3
    volumes:
      - /Users/robin/learning/docker/mongo_replset/mongo_3/mongod.conf:/etc/mongod.conf
      - /Users/robin/learning/docker/mongo_replset/mongo_3/initdb.d/:/docker-entrypoint-initdb.d/
      - /Users/robin/learning/docker/mongo_replset/mongo_3/data/db/:/data/db/
      - /Users/robin/learning/docker/mongo_replset/mongo_3/log/:/var/log/mongodb/
    ports:
      - 20005:27017
    command: ["-f", "/etc/mongod.conf","--replSet", "rs_mongo"]
    network_mode: mongo_net

Step 4: create mongod.conf

$  code .

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /data/db
  journal:
    enabled: true
  engine:  wiredTiger

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1

Step 5: Spin-up replicaset containers

$ docker compose up -d
[+] Running 3/3
 ⠿ Container mongo_2  Created                                                                                                                                   0.2s
 ⠿ Container mongo_1  Created                                                                                                                                     0.2s
 ⠿ Container mongo_3  Created

Step 6: Initiate replicaset

$ docker exec -it mongo_1 bash

root@mongo_1:/# mongo
rs_mongo:SECONDARY> rs.initiate(
   {
      _id: “rs_mongo”,
      version: 1,
      members: [
         { _id: 0, host : “mongo_1:27017” },
         { _id: 1, host : “mongo_2:27017” },
         { _id: 2, host : “mongo_3:27017” }
      ]
   }
)

rs_mongo:SECONDARY> db.isMaster() 
{
    "topologyVersion" : {
        "processId" : ObjectId("614615744d54c08963ef67f6"),
        "counter" : NumberLong(6)
    },
    "hosts" : [
        "mongo_1:27017",
        "mongo_2:27017",
        "mongo_3:27017"
    ],
    "setName" : "rs_mongo",
    "setVersion" : 1,
    "ismaster" : true,
    "secondary" : false,
    "primary" : "mongo_2:27017",
    "me" : "mongo_2:27017",

The article is divided in two parts, the first part is setting up the standalone MongoDB container and second part is setting up and grouping MongoDB containers as member of replica set with Docker.

Let’s get started.

System Configuration

To run this setup, Docker Engine is required to be installed on the system. Follow the official documentation to setup Docker Engine on your system.

:::caution

The steps and configuration for both standalone and replica set is not to be used for production deployment. The intended use is only for setting up a environment to support learning of MongoDB.

:::

Standalone MongoDB Setup

• Pull the Docker MongoDB official image from Docker Hub. The following code snippet demonstrates pulling the docker MongoDB 4.4.9 release. To pull the MongoDB 5.0 latest release replace :4.4.9-rc0 with :latest tag

$ docker pull mongo:4.4.9-rc0

• To check if the the image pull from Docker Hub was successful

$ docker images                                                   
REPOSITORY   TAG         IMAGE ID       CREATED       SIZE
mongo        4.4.9-rc0   24599d6cde30   9 days ago    413MB
mongo        latest      31299b956c79   10 days ago   642MB

• Lets start first standalone container – the below command starts MongoDB docker container with name mongo_449 in detached mode using the 4.4.9-rc0 image

$ docker run --name mongo_449 -d mongo:4.4.9-rc0

• List the container status and health by executing

$ docker container ls -a

CONTAINER ID   IMAGE          COMMAND                  CREATED       STATUS                      PORTS       NAMES
96e64ec525a2   24599d6cde30   "docker-entrypoint.s…"   2 hours ago   Up 33 minutes               27017/tcp   mongo_449

• To run a command inside the container
  • docker exec: interact with containers (running/up mode)
  • -i : interactive STDIN open even if not attached to the container
  • -t: pseudo TTY

• Connect to MongoDB daemon

root@96e64ec525a2:/# mongo

MongoDB shell version v4.4.9-rc0
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("ac624a79-908b-4580-90ae-22d0a7aee07a") }
MongoDB server version: 4.4.9-rc0

• Install utilities. The utilities ping, systemctl, sudo installed in the containers can be used for troubleshooting during the setup of Docker containers.

root@96e64ec525a2:/# apt-get install iputils-ping̵
root@96e64ec525a2:/# apt-get install sudo 
root@96e64ec525a2:/# apt-get install systemctl

This finishes the setup of standalone MongoDB Container. Now let’s look at ReplicaSet setup.

Creating MongoDB ReplicaSet using Docker

A replica set consists of a primary node together with two or more secondary nodes. It is recommended to group three or more nodes, with an odd number of total nodes. The primary node accepts all the write requests which are propagated synchronously or asynchronously to the secondary nodes. Below are the steps required to complete the replica set setup using Docker.

Create a new network(bridge) within Docker. The replica set containers will be mapped to the new network.

$ docker network create mongo_net
$ docker network inspect mongo_net                       
[
    {
        "Name": "mongo_net",
        "Id": "e2567806642a9245436371a9b9904c71fadae969fbd11a7bb8203e07976b1b2a",
        "Created": "2021-09-11T00:36:33.989688708Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
...
]

• Start 3 containers – Primary Secondary Secondary
  • Break down of parameters docker run : start a new container
    • -d : run the container in detached mode
    • -p 20001:27017 publish container port to the host and bind 27017 to 20001 on the host. This is useful if connecting mongo client like mongosh to container
    • --name : name of the mongo container
    • -- network : connect to user created network mongo_net
    • mongo:4.4.9-rc0 : Docker MongoDB image
    • mongod --replSet rs_mongo : run the mongod daemon and add the container to replica set name rs_mongo

$ docker run -d -p 20001:27017 --name mongo1 --network mongo_net mongo:4.4.9-rc0 mongod --replSet rs_mongo
$ docker run -d -p 20002:27017 --name mongo2 --network mongo_net mongo:4.4.9-rc0 mongod --replSet rs_mongo
$ docker run -d -p 20003:27017 --name mongo3 --network mongo_net mongo:4.4.9-rc0 mongod --replSet rs_mongo

• Set up Replica set. Connect to one of the containers and run the below commands. The container that receives the initiate will pass on the configuration to other containers assigned as members.

rs_mongo [direct: primary] test_2> config = {
      "_id" : "rs_mongo",
      "members" : [
          {
              "_id" : 0,
              "host" : "mongo1:27017"
          },
          {
              "_id" : 1,
              "host" : "mongo2:27017"
          },
          {
              "_id" : 2,
              "host" : "mongo3:27017"
          }
      ]
  }

rs_mongo [direct: primary] admin> rs.initiate(config)

//Insert test data

rs_mongo [direct: primary] admin> use test_2
rs_mongo [direct: primary] test_2> db.employees.insert({name: "robin")

//To read queries on secondary run setReadPref. 
rs_mongo [direct: secondary] test_2>db.getMongo().setReadPref('secondary')

rs_mongo [direct: secondary] test_2> db.employees.find()
[
  { _id: ObjectId("613c99801ea796508e3c73f5"), name: 'robin' }
]

• Validate Replica Set Configuration

rs_mongo [direct: primary] test_2> db.printReplicationInfo()

configured oplog size
'557174 MB'
---
log length start to end
'71372 secs (19.83 hrs)'
---
oplog first event time
'Sat Sep 11 2021 15:47:21 GMT+0530 (India Standard Time)'
---
oplog last event time
'Sun Sep 12 2021 11:36:53 GMT+0530 (India Standard Time)'
---
now
'Sun Sep 12 2021 11:36:54 GMT+0530 (India Standard Time)'


rs_mongo [direct: primary] test_2> rs.conf()
{
  _id: 'rs_mongo',
  version: 1,
  term: 1,
  protocolVersion: Long("1"),
  writeConcernMajorityJournalDefault: true,
  members: [
    {
      _id: 0,
      host: 'mongo1:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      slaveDelay: Long("0"),
      votes: 1
    },
    {
      _id: 1,
      host: 'mongo2:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      slaveDelay: Long("0"),
      votes: 1
    },
    {
      _id: 2,
      host: 'mongo3:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      slaveDelay: Long("0"),
      votes: 1
    }

That concludes this article.