Github

Github is a company that runs servers to manage software source code using git tool.

It has free public repositories and non-free private repositories.

With github we do this where the domain name is github.com:

# clone a repository
git clone [email protected]/k-five/private-git-server
# or
git remote add origin [email protected]/k-five/private-git-server


# for push 
git push -u origin master

# git remote -v
origin [email protected]/k-five/private-git-server.git (fetch)
origin [email protected]/k-five/private-git-server.git (push)

Gitlab

Gitlab is the same somehow.

It offers both public and private repositories for free.

With gitlab we do this where the domain name is gitlab.com

# clone a repository
git clone [email protected]/k-five/private-git-server
# or
git remote add origin [email protected]/k-five/private-git-server

# for push 
git push -u origin master

# git remote -v
origin [email protected]/k-five/private-git-server.git (fetch)
origin [email protected]/k-five/private-git-server.git (push)

Private Git Server

Private git server will be our own server that uses git tool as our version control system.

Although it is possible to have public repositories with a private-git-server, but the main purpose is to have private repositories.

With our server we do this where the domain name is our own at private-git-server.ir

# clone a repository
git clone [email protected]:private-git-server.git
# or
git remote add origin [email protected]:private-git-server.git

# for push 
git push -u origin master

# git remote -v 
# for our private git server
origin	[email protected]:private-git-server.git (fetch)
origin	[email protected]:private-git-server.git (push)

It is not accessible to outside world unless we have access to the server using SSH.

But for testing purposes your can clone it using git protocol.

# test if the repository is available
 git ls-remote  git://95.216.12.6/private-git-server
5050fde5f52330592fc6a84c83b0fa6149b02026	HEAD
5050fde5f52330592fc6a84c83b0fa6149b02026	refs/heads/master

# clone the repository
# this is publicly available just for TEST
git clone  git://95.216.12.6/private-git-server
Cloning into 'private-git-server'...
remote: Counting objects: 112, done.
remote: Compressing objects: 100% (103/103), done.
remote: Total 112 (delta 31), reused 0 (delta 0)
Receiving objects: 100% (112/112), 218.64 KiB | 387.00 KiB/s, done.
Resolving deltas: 100% (31/31), done.
Checking connectivity... done.

Architecture


Here is the top level architecture for our private git server.

  • end users: our customers, etc
  • PS: Production Server
  • PGS: Private Git Server
  • CIS: Continuous Integration Server
  • githb: Git Server for free public repositories
  • gitlab: Git Server for free public and private repository
  • dev team: the developers who code
  • other: any other contributors to our public repositories

Node that in this architecture using gitlab or github are completely optional. We do not have to use them because we do have shared repositories on our private git server.

The usage of gitlab. github, others services is for their rich Web Interface they provide for code, commit, issue review, etc.

Private Git Server Animation Author: Shakiba Moshiri end users end users end users PS production server change notification pull clone PGS CIS dev team push pull, clone email, sms others clone pull push Continuous Integration Continuous Deployment Continuous Delivery Private Git Server CI Server github public push web-hook notification gitlab private push web-hook notification Creator & Designer: Shakiba Moshiri 2019 All Right Reserved

Workflow


  1. Our teams, developers, commit new code and push to the private git server.
  2. Private git server receives new code and merges new code into old one.
  3. If merge succeeded, it will be pushed to gitlab (private) repository, or github (public) repository.
  4. After successful merge, if there is a test, it cab be run.
  5. After successful merge, if there is a build, it cab be run.
  6. Any kinds of notification can be set here, e.g Email, Text, etc
  7. Finally when there is no error, a change/update notification is delivered to production server to tell, to synchronize itself with last changes.

Use case


The best use case for this architecture is for countries e.g. Iran that have been blocked by gitlab, etc and developers in these countries have to use a VPN, or setup a proxy for git, or other dirty solutions.

None of the solutions I have tired myself and have seen that others tired are the correct and the right ones.

So I came up with this idea of having a Private Git Sever outside of a blocked country, like me, since I live in Iran.

For testing purpose you can contribute if you liked; so just notify me.

Monitoring


Here is quick review of kinds of monitoring we can have.

Web based monitoring

Using HTML5 Web Socket API we can have a real-time monitoring using our web browsers.

Here is an example rtsm.ir.

Git based monitoring

My innovation.

Using the git itself, we can have embedded monitoring as you can see in below screenshots.

It is the fastest and most effective way I could come up with. We can embed anything we like.

It shows that both push, and merge succeeded.

pgs-merge-succeeded.png

What if it did not succeed?

The output of git is embedded when failed. For example here file-5, file-6 on the server has changed without committing. And git cannot override them. So it gives us the error instantly..

I am one commit behind, so I try pushing ...

pgs-merge-failed.png

Is not it amazing?

Phone based monitoring

If we like to receive a kind of notification on our phone, we can setup Text Message Notification.

This is an asynchronous notification that unlike emailing we do not have to check our email periodically.

For example using ndoe.js, or bash API like so:

#!/bin/bash

# Get the API KEY from your xxxxxxxxx.com account
readonly API_KEY='our-api-key';

to="$1" 
from="$2"
text="$3" 

echo "$(date) ${text} ${from} $to" &>> /tmp/sms.log

curl -X POST "http://api.xxxxxxxxx.com/v2/sms/send/simple" \
	-H "apikey: ${API_KEY}" \
	-H 'cache-control: no-cache' \
	-H 'content-type: application/x-www-form-urlencoded' \
	-d "message=${text}&linenumber=${from}&receptor=${to}" &>> /tmp/sms.log

SSH based monitoring

My innovation.

Using SSH and keeping the connection alive helps us to have a secure, fast, real-time, and amazing monitoring.

The interesting part of using SSH is that we can redirect the output to our psuedo-terminal so we can have a single terminal for both pushing and monitoring.

Here is screenshot that after pushing to the server, the result has been redirected to me.

ssh-monitoring.png
  • I push to github
  • github notifies jsfun.ir
  • jsfun.ir does a merge
  • merge result comes from server to my local machine using SSH
  • then is redirected to the terminal I pushed from!

All happens within 2 or 3 seconds.

Third-party services for monitoring

There are also services that using their APIs we can do monitoring e.g Slack, etc

Task and Automation


Four tasks:

  • synchronous
    1. push
    2. merge
  • asynchronous
    1. git push to github
    2. git push to gitlab

This shot indicates that push and merge have succeeded, and push to github, gitlab will be run in background.

run-task.png

And this one shows the log on the server.

run-task-log.png

Contact Me


If you wanted a fully automated pipeline and solve the problem of your access to gitlab, in the right way, notify me via:

  • email me: n e t @ s h a k i b a . i t

to schedule a conversion over Skype. If you already have Skype, you can find me with Name / ID:

  • Name: "Shakiba Moshiri"
  • ID: +98 930 781 8080

Also if you had any other questions or good ideas, feel free to tell them to me so we can learn from each other or work together.

NOTE

This homepage source code as a repository for test is available on: