Open in app

Sign in

Write

Sign in

Mohammad Meskarian
7 min readSep 28, 2020

--

MongoDB Shard Cluster

In this tutorial, I’m going to show you how to set up a MongoDB cluster (including mongo-shard, mongo-config, and mongos instances) on a RedHat based OS (centos 7), how to configure each part.

Mongo Shard Schema

For the sake of this tutorial, I’m using 20 centos 7 nodes. The bare minimum requirement for this installation is at least 3 mongo-config servers (consider using at least 3 nodes for production use cases. ), Two shards (each shard including 5 replicas = I’m going to replicate data 5 times). And at last, you need as many mongos instances as you want.

Along with the post, I will recommend security considerations. It is a best practice to use them in production cases.

You can skip this step if you don’t want to disable transparent huge pages.

Step 0) Precautions:

To avoid facing the popular WARNING of mongo:

WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'

I used this link. Which is pretty straightforward:
According to the MongoDB docs:

Transparent Huge Pages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amounts of memory by using larger memory pages.

However, database workloads often perform poorly with THP enabled, because they tend to have sparse rather than contiguous memory access patterns. When running MongoDB on Linux, THP should be disabled for best performance.

To ensure that THP is disabled before mongod starts, you should create a service file for your platform’s initialization system that disables THP at boot. Instructions are provided below for both the systemd and the System V init initialization systems.

So I’ll create a service file and put it inside the “systemd” location:

Create the following file at

/etc/systemd/system/disable-transparent-huge-pages.service:

[Unit]
Description=Disable Transparent Huge Pages (THP)
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=mongod.service[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo never | tee /sys/kernel/mm/transparent_hugepage/enabled > /dev/null'[Install]
WantedBy=basic.target

Reload unit files:

systemctl daemon-reload

Start and enable the service at the startup:

systemctl enable --now disable-transparent-huge-pages

Check the service:

systemctl status disable-transparent-huge-pages

Verify that THP has successfully been set to [never] by running the following command:

cat /sys/kernel/mm/transparent_hugepage/enabled

Step 1) Disable SELinux and Configure Hosts
on all servers, change selinux policy to disabled :

vim /etc/sysconfig/selinux
SELINUX=disabled

Define all of your hosts in “/etc/hosts”. You can use FQDN records as well.
you can change the hostname of your server by using “hostnamectl” command:

hostnamectl set-hostname mongo-config01
...# verify
hostname

edit /etc/hosts:

vim /etc/hosts192.168.109.130 mongo-config01
192.168.109.131 mongo-config02
192.168.109.132 mongo-config03
192.168.109.133 mongo-config04
192.168.109.134 mongo-config05
192.168.109.135 mongo-sh01
192.168.109.136 mongo-sh02
192.168.109.137 mongo-sh03
192.168.109.138 mongo-sh04
192.168.109.139 mongo-sh05
192.168.109.140 mongo-sh06
192.168.109.141 mongo-sh07
192.168.109.142 mongo-sh08
192.168.109.143 mongo-sh09
192.168.109.144 mongo-sh10
192.168.109.145 mongos-01
192.168.109.146 mongos-02
192.168.109.147 mongos-03
192.168.109.148 mongos-04
192.168.109.149 mongos-05

Step 2) Install MongoDB on all instances:
Add the desired MongoDB version to the list of repositories on your server.
I’m using version 4.4 of the MongoDB:

vim /etc/yum.repos.d/mongodb-org-4.4.repo[mongodb-org-4.4]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/4.4/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-4.4.asc

Install and verify:

yum install -y mongodb-org
rpm -qa | grep mongo
mongod --version

Alternatively, to install a specific release of MongoDB, specify each component package individually and append the version number to the package name, as in the following example:

yum install -y mongodb-org-4.4.1 mongodb-org-server-4.4.1 mongodb-org-shell-4.4.1 mongodb-org-mongos-4.4.1 mongodb-org-tools-4.4.1

Step 3) Create Config Server Replica Set

systemctl stop mongod
mkdir -p /data/db
chown -R mongod:mongod /data/db

Only on config servers:
Remember to replace your IP address in the <SERVER_IP>:

## Config Servers mongo.conf
mv /etc/mongod.conf /etc/mongo.backup
vim /etc/mongod.conf# log
systemLog:
 destination: file
 logAppend: true
 logRotate: rename
 path: /var/log/mongodb/mongod.log# data
storage:
 dbPath: /data/db
 journal:
  enabled: true# process
processManagement:
 fork: true # fork and run in background
 pidFilePath: /var/run/mongodb/mongod.pid
 timeZoneInfo: /usr/share/zoneinfo# network Interface and port
net:
 port: 27017
 bindIp: 127.0.0.1, <SERVER_IP>#security:
# authorization: enabled
# keyFile: /data/mongodb.key# replica_name , you can use any desired name.
replication:
 replSetName: "replconfig01"# role in the cluster. This will define whether it's a config server # or a shard server, in config servers choose "configsvr".
 
sharding:
 clusterRole: "configsvr"
##

Start the instance:

chmod 644 /etc/mongod.conf
mongod --config /etc/mongod.conf

Output:
about to fork child process, waiting until server is ready for connections.
forked process: 4566
child process started successfully, parent exiting

# check port 27017
netstat -plntu | grep 27017

on mongo-config01:

## mongo shell
mongo --host mongo-config01 --port 27017

Initiate the replica set name with all configsvr member using the query below:

rs.initiate(
 {
 _id: "replconfig01",
 configsvr: true,
 members: [
 { _id : 0, host : "mongo-config01:27017" },
 { _id : 1, host : "mongo-config02:27017" },
 { _id : 2, host : "mongo-config03:27017" },
 { _id : 3, host : "mongo-config04:27017" },
 { _id : 4, host : "mongo-config05:27017" } 
 ]
 }
)

If you see the output:

‘{ “ok” : 1 }’,

it means the configsvr is configured with a replica set.
And you can see which node is master and which node is secondary.

rs.isMaster()
rs.status()

Add selinux permission (if selinux activated)

semanage port -a -t mongod_port_t -p tcp 27017

Step 4) Configure Shard Servers 1:

ON ALL SHARD SERVERS:

mv /etc/mongod.conf /etc/mongo.backup
systemctl stop mongod
mkdir -p /data/db
chown -R mongod:mongod /data/db

ON SHARD SERVER 1:

meaning:

192.168.109.135 mongo-sh01
192.168.109.136 mongo-sh02
192.168.109.137 mongo-sh03
192.168.109.138 mongo-sh04
192.168.109.139 mongo-sh05

Remember to replace your IP address in the <SERVER_IP>:

vim /etc/mongod.conf# log
systemLog:
 destination: file
 logAppend: true
 logRotate: rename
 path: /var/log/mongodb/mongod.log# data
storage:
 dbPath: /data/db
 journal:
  enabled: true# process
processManagement:
 fork: true # fork and run in background
 pidFilePath: /var/run/mongodb/mongod.pid
 timeZoneInfo: /usr/share/zoneinfo# network Interface and port
net:
 port: 27017
 bindIp: 127.0.0.1, <SERVER_IP>#security:
# authorization: enabled
# keyFile: /data/mongodb.key# replica_name SHARD1
replication:
 replSetName: "shardreplica01"# role in the cluster
sharding:
 clusterRole: "shardsvr"

ON SHARD SERVER 2:

meaning:

192.168.109.140 mongo-sh06
192.168.109.141 mongo-sh07
192.168.109.142 mongo-sh08
192.168.109.143 mongo-sh09
192.168.109.144 mongo-sh10

Remember to replace your IP address in the <SERVER_IP>:

vim /etc/mongod.conf
# log
systemLog:
 destination: file
 logAppend: true
 logRotate: rename
 path: /var/log/mongodb/mongod.log# data
storage:
 dbPath: /data/db
 journal:
  enabled: true# process
processManagement:
 fork: true # fork and run in background
 pidFilePath: /var/run/mongodb/mongod.pid
 timeZoneInfo: /usr/share/zoneinfo# network Interface and port
net:
 port: 27017
 bindIp: 127.0.0.1, <SERVER_IP>#security:
# authorization: enabled
# keyFile: /data/mongodb.key# replica_name SHARD2
replication:
 replSetName: "shardreplica02"# role in the cluster
sharding:
 clusterRole: "shardsvr"

Start mongod instance:

chmod 644 /etc/mongod.conf
mongod --config /etc/mongod.conf

SHARD configuration (shard1):

on shard1 (5 servers, with replication 5):

ssh root@mongo-sh01
mongo --host mongo-sh01 --port 27017

Initiate the first shard:

rs.initiate(
 {
 _id : "shardreplica01",
 members: [
 { _id : 0, host : "mongo-sh01:27017" },
 { _id : 1, host : "mongo-sh02:27017" },
 { _id : 2, host : "mongo-sh03:27017" }, 
 { _id : 3, host : "mongo-sh04:27017" },
 { _id : 4, host : "mongo-sh05:27017" } 
 ]
 }
 )

Output:

{
 "ok" : 1,
 "$clusterTime" : {
 "clusterTime" : Timestamp(152298186, 1),
 "signature" : {
 "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
 "keyId" : NumberLong(0)
 }
 },
 "operationTime" : Timestamp(1522981861, 1)
}

SHARD configuration (shard2):
On shard2 (5 servers, with replication 5):

ssh root@mongo-sh06
mongo --host mongo-sh06 --port 27017
rs.initiate(
 {
 _id : "shardreplica02",
 members: [
 { _id : 0, host : "mongo-sh06:27017" },
 { _id : 1, host : "mongo-sh07:27017" },
 { _id : 2, host : "mongo-sh08:27017" }, 
 { _id : 3, host : "mongo-sh09:27017" },
 { _id : 4, host : "mongo-sh10:27017" } 
 ]
 }
 )

5) Configure mongos (mongodb query router):

setenforce 0
mv /etc/mongod.conf /etc/mongo.backup
systemctl stop mongod ; systemctl disable mongod

Remember to change the value of <SERVER_IP>:

vim /etc/mongos.conf# log
systemLog:
 destination: file
 logAppend: true
 logRotate: rename
 path: /var/log/mongos/mongos.logprocessManagement:
 fork: true # fork and run in background
 pidFilePath: /var/run/mongodb/mongod.pid
 timeZoneInfo: /usr/share/zoneinfo# network interfaces
net:
 port: 27017
 bindIp: 127.0.0.1, <SERVER_IP>sharding:
 configDB: replconfig01/mongo-config01:27017,mongo-config02:27017,mongo-config03:27017,mongo-config04:27017,mongo-config05:27017

Configure Log file:

mkdir -p /var/log/mongos/
touch /var/log/mongos/mongos.log
chown mongod:mongod /var/log/mongos
chown mongod:mongod /var/log/mongos/*

Start mongos query router:

mongos --config /etc/mongos.conf

Add mongos to the systemd unit:

vim /etc/systemd/system/multi-user.target.wants/mongos.service
[Unit]
Description=Mongo Cluster Router
After=network.target[Service]
User=mongod
Group=mongodExecStart=/usr/bin/mongos --config /etc/mongos.confLimitFSIZE=infinity
LimitCPU=infinity
LimitAS=infinityLimitNOFILE=64000
LimitNPROC=64000TasksMax=infinity
TasksAccounting=false[Install]
WantedBy=multi-user.target
##

Reload systemd daemons:

systemctl daemon-reload
systemctl start mongos
systemctl status mongos

Start and config mongos:

mongo --host mongos-01 --port 27017

For ‘shardreplica01’ instances:

sh.addShard( "shardreplica01/mongo-sh01:27017,mongo-sh02:27017,mongo-sh03:27017,mongo-sh04:27017,mongo-sh05:27017")

For ‘shardreplica02’ instances:

sh.addShard( "shardreplica02/mongo-sh06:27017,mongo-sh07:27017,mongo-sh08:27017,mongo-sh09:27017,mongo-sh10:27017")

See the mongos status:

sh.status()

On mongos server test the sharding:

mongo --host mongos-01 --port 27017
use taxidb
# enable sharding for the taxidb
sh.enableSharding("taxidb")

Check data location:

sh.status()databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shardreplica01 1

We see that data is on shardreplica01,
Next, include brand new collections on the database with sharding help. We are going to include a collection that is new ‘stack’ with shard collection ‘name’, and then see database and collections status.

sh.shardCollection("taxidb.newyork", {"myshardkey":1})
# see where is the data:
sh.status()

Now collections ‘NewTest’ with shard collection ‘name’ is added.
Add papers to collections ‘NewTest’.
Now insert the papers on collections. We must include the ‘shard key’.( when we add documents to the collection on the sharded cluster)

In this example, I’m using a key that is shard name, once we included

db.newyork.save({"name": "taxidb newyork","Union Members": ["John", "Jack", "Jason", "Gerald"],})# exptected output:
WriteResult({ “nInserted” : 1 })

find the data and then use remote mongo shell:
From mongos-01 instance to the primary shard02

mongo --host mongo-sh06 --port 27017
show dbs
use 
db.newyork.find()
## if you see the data then it’s done.

on all mongo servers:

semanage port -a -t mongod_port_t -p tcp 27017

If you need to update the document name:

db.newyork.update({"name":"taxidb new"})

Reference: https://docs.mongodb.com/manual/tutorial/deploy-shard-cluster/

Mongodb
Shard
Database
Desgin
Linux

--

--

Mohammad Meskarian

Written by Mohammad Meskarian

4 Followers

Help

Status

About

Careers

Press

Blog

Privacy

Terms

Text to speech

Teams