How to setup a virtual machine

How to setup a virtual machine ?

This post covers how to setup VirtualBox , VMware Workstation, VMware Player and Vagrant applications and how to setup guest operating system.

Here, I’ve selected Windows 10 Home as Host OS and Ubunbu 16.04 LTS Xenial as guest OS.

Let’s first understand some basic terminologies regarding Virtualization.

Virtualization :
Virtualization is the creation of a virtual (rather than actual) version of something, such as an operating system, a server, a storage device or network resources.

HostedVirtualizationSmall_20090325191230

Hypervisor :
A hypervisor or virtual machine monitor (VMM) is a piece of computer software, firmware or hardware that creates and runs virtual machines.

virtualization

A computer on which a hypervisor is running one or more virtual machines is defined as a host machine. Each virtual machine is called a guest machine. The hypervisor presents the guest operating systems with a virtual operating platform and manages the execution of the guest operating systems. Multiple instances of a variety of operating systems may share the virtualized hardware resources.

Type-1, native or bare-metal hypervisors
These hypervisors run directly on the host’s hardware to control the hardware and to manage guest operating systems. For this reason, they are sometimes called bare metal hypervisors.

Type-2 or hosted hypervisors
These hypervisors run on a conventional operating system just as other computer programs do.
A guest operating system runs as a process on the host. Type-2 hypervisors abstract guest operating systems
from the host operating system. VMware Workstation, VMware Player, VirtualBox and QEMU are examples of type-2 hypervisors.

400px-Hyperviseur

VirtualBox :

VirtualBox is a powerful x86 and AMD64/Intel64 virtualization product for enterprise as well as home use. It is freely available as Open Source Software under the terms of the GNU General Public License (GPL) version 2.

Presently, VirtualBox runs on Windows, Linux, Macintosh, and Solaris hosts and supports a large number of guest operating systems including but not limited to Windows (NT 4.0, 2000, XP, Server 2003, Vista, Windows 7, Windows 8, Windows 10), DOS/Windows 3.x, Linux (2.4, 2.6, 3.x and 4.x), Solaris and OpenSolaris, OS/2, and OpenBSD.

You can download it from here VirtualBox

There are two options to setup guest OS,
1) You need Ubuntu ISO image to set it up new guest OS machine. It can be downloaded from here ubuntu-16.04-desktop-amd64.iso.
2) You can directly download ready box from osboxs.

I’ve chosen to setup new machine using ISO image.

Installation setups are given on their website which are pretty simple. Installation steps has chosen default setting while allocation size/space to guest OS.
You need to make sure that your guest machine has sufficient space both for application and data which you will keeping on it.

Follow these steps to setup virtual machine,

Virtualbox Setting Up a New Virtual Machine

You can also share folder between Host and guest machine using SharedFolders.

Guest Additions setup can be found here, Guest Additions.

Following are important steps for setting up shared folder,

sudo apt-get install build-essential
sudo apt-get install linux-headers-`uname -r`

sharename="shared_folder"
sudo mkdir /mnt/$sharename
sudo chmod 777 /mnt/$sharename
sudo mount -t vboxsf -o uid=1000,gid=1000 $sharename /mnt/$sharename
ln -s /mnt/$sharename $HOME/Desktop/$sharename

For me, it got mounted on /media/sf_share path for some reason.

If google chrome in guest OS (Ubuntu) is not working properly then Disable 3D-acceleration in guest OS setting on virtual box.

Resize Virtualbox Image :

1) Backup VDI image:

Take a backup of you image by coping ‘.vdi’ file to somewhere else. You may need it to restore your system in case of any data lose.

path_to_folder\VirtualBox VMs\Xenial64\Xenial64.vdi

2) Extent VDI image size:

"C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" modifyhd "path_to_folder\VirtualBox VMs\Xenial64\Xenial64.vdi" --resize 30000
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%

Above command will extent VDI size of you image but you will still need to allocate that space to guest OS partition manually.

3) Allocate space to partition:

You can resize your existing guest OS parting using gparted.

Download gparted from this link, (Choose correct 32bit/64bit)
gparted

Boot your virtual machine using above gparted ISO.

NOTE: If gparted gui not working or you it is getting hanged in between.
Go to VM Setting->Systems and checked the “Enable EFI (special OSes only)” extended feature.

Now you should see gparted GUI.

You can see /dev/sda1, /dev/sda2 (linux-swap) partitions on screen. You can easily resize /dev/sda2 (linux-swap) partition by selecting it, choosing “Resize/Move” and applying using “Apply” button.

But, If you wish to extent /dev/sda1 partition then you will need to delete linux-swap and /dev/sda2 partition. Now It will allow you to resize /dev/sda1 partiontion. Set it whatever size you want. Also keep

some 3-4GB free for linux-swap.

Now, re-add your Linux Swap Space. Press on the unallocated space and right-click to “Created new Partition” -> choosing “Create as: Extended partition” .

Press “+Add” and right-click the new “unallocated” to Create a new partition. Choose as “Create as: Logical Partition” and underneath “File System: linux-swap”.

Now press “+Add” and then “Apply” in the main window. Hopefully all changes are successfully applied.

You can now safely shut down this Live CD Virtual Machine.

You can now check disk space again.

#df -h

VMware Workstation / VMware Player :

VMware virtualizes computing, from the data center to the cloud to mobile devices, to help our customers be more agile, responsive, and profitable.

VMware Workstation and VMware Player transforms the way technical professionals develop, test, demonstrate and deploy software by running multiple x86-based operating systems simultaneously on the same PC.

You can download it from here Vmware

Most of installation steps are identical to for both of them. There are few differences which can be found here on compare page.

Follow these steps to setup virtual machine,

VMware Workstation Setting Up a New Virtual Machine

Here also you will need some shared folder to sync data between Host and Guest machine.

Install vmware-tools as follows,

sudo apt-get install build-essential
sudo apt-get install linux-headers-`uname -r`
cp /cdrom/VM*.gz /tmp/
cd /tmp
tar xvzf VM*.gz
cd vmware-tools*
sudo ./vmware-install.pl

NOTE : if you got,
The package that need to be removed,
open-vm-tools

sudo apt-get remove open-vm-tools;

and try running above command again.

Hit enter for all defaults.

Details are given here, Installing VMware tools on an Ubuntu guest

Once it is done, follow steps given as below to setup shared folder,

Create a New Shared Virtual Machine

Now, your shared folder should be ready on following path,

ls /mnt/hgfs/

Vagrant :

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.

To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.

You can get it here, Vagrant

To set it up on ubuntu follow this Setup vagrant on ubuntu

You can find ready boxes here, hashicorp.

Now we can just start your virtual machine as follows,

C:\HashiCorp\Vagrant\bin>vagrant init boxcutter/ubuntu1604
C:\HashiCorp\Vagrant\bin>vagrant up --provider virtualbox

If you are using Window as Host OS and Ubuntu Server as guest OS then While setup you will get following details,

Username : ubuntu
PrivateKey : Drive:\HashiCorp\Vagrant\bin\.vagrant\machines\default\virtualbox\private_key

As Window doesn’t has ssh client by default. You will need ssh client like ‘putty’ to ssh to guest OS Ubuntu server.

Putty can be downloaded from here, putty.

Also download PuTTYgen.

- PuTTY – Client to for managing SSH sessions
- PuTTYgen – Tool for managing and creating SSH key pairs

Load Vagrant private key in PuTTYgen, set key passphrase (remember it. Will need later) and click on ‘Save private key’ button to Generate Putty(.ppk) ‘Private’.
Use Putty private key (.ppk) to set it under SSH-Auth section in Putty. You can also set default auth username under Putty Connection-Data section.

Now you should be able to ssh to guest Ubuntu server OS.

You can also setup shared folder on Vagrant. Steps are given here , Shared folder.

Now you can setup as many as guest OS using above applications.

Thanks.

Posted in Uncategorized, Virtualization | Tagged , , | Leave a comment

Cross-Origin Resource Sharing

Cross-Origin Resource Sharing:

Cross-Origin Resource Sharing (CORS) is a specification that enables truly open access across domain-boundaries.
If you serve public content, please consider using CORS to open it up for universal JavaScript/browser access.
Usually web browsers forbids cross-domain requests, due the same origin security policy.
Cross-origin resource sharing (CORS) is a technique that allow servers to serve resources to permitted origin domains
by adding HTTP headers to the server who are respected from web browsers.

Web Server Settings:

You can send the response header with PHP like,

header("Access-Control-Allow-Origin: *");
header("Access-Control-Allow-Headers: origin, x-requested-with, content-type");
header("Access-Control-Allow-Methods: GET, OPTIONS");

Or Set Header using Web Server. I am using Apache Web Server for this tutorial.

To add the CORS authorization to the header using Apache, simply add headers inside either
the <Directory>, <Location>, <Files> or <VirtualHost> sections of your server config
(usually located in a *.conf file, such as httpd.conf or apache.conf), or within a .htaccess file.

<IfModule mod_headers.c>
Header add Access-Control-Allow-Origin "*"
Header add Access-Control-Allow-Headers "origin, x-requested-with, content-type"
Header add Access-Control-Allow-Methods "GET, OPTIONS"
</IfModule>

Adding “*” in header will make your url public. Any one can request data from that url.

If you want to restrict it to one/more domains as follows

1) Only one domain “http://search.mysite.com/”.

Header add Access-Control-Allow-Origin "http://search.mysite.com/"

2) Multiple sub-domain

SetEnvIf Origin "^(.*\.mysite\.com)$" ORIGIN_SUB_DOMAIN=$1
Header set Access-Control-Allow-Origin "%{ORIGIN_SUB_DOMAIN}e" env=ORIGIN_SUB_DOMAIN

3) Multiple Other domain.

SetEnvIf Origin "^http(s)?://(.+\.)?(mysite\.org|mysite\.com)$" ORIGIN_DOMAIN=$0
Header always set Access-Control-Allow-Origin %{ORIGIN_DOMAIN}e env=ORIGIN_DOMAIN

File can be found at following location,
1) /etc/apache2/sites-available/mysite.conf
2) /etc/apache2/apache2.conf
3) /var/www/mywebsite/.htaccess

To ensure that your changes are correct, it is strongly reccomended that you use
#apachectl -t

to check your configuration changes for errors. After this passes, you may need to reload Apache to make sure your changes are applied by running the command

#sudo service apache2 reload
or
#apachectl -k graceful

Altering headers requires the use of mod_headers. Mod_headers is enabled by default in Apache, however, you may want to ensure it’s enabled by run
#a2enmod headers

Note: you can also use add rather than set, but be aware that add can add the header multiple times, so it’s likely safer to use set.

Web Browser/Client Code:

Include following scripts,

<link href="//code.jquery.com/ui/1.11.4/themes/smoothness/jquery-ui.css" rel="stylesheet" />
<script type="text/javascript" src="//code.jquery.com/jquery-1.10.2.js">
</script><script type="text/javascript" src="//code.jquery.com/ui/1.11.4/jquery-ui.js"></script>

and add following code at the end of your page.


$("#autocomplete").autocomplete({
autoFocus: true,
minLength: 3,
source: function (request, response) {
$.ajax({
url: "http://www.mysite.com/url",
data: 'query=' + request.term,
crossDomain: true,
type: "GET",
contentType: "application/json; charset=utf-8",
async: false,
success: function (data, status) {
if (data.content.error != null) {
alert(data.content.error);
} else {
var x;
var item = [];
for (x in data.content.data) {
item[x] = data.content.data[x].url;
}
response(item);
}
},
error: function (err) {
alert(err);
}
});
},
select: function (event, ui) {
val = $("#autocomplete").val();
window.open(ui.item.value);
$("#autocomplete").val(val);
return false;
}
});

If you want to prevent Apache from executing your code when the browser sends a HTTP OPTIONS Prefight request , returning 200 OK, Add following rewrite rule in .htaccess,

<IfModule mod_rewrite.c>
RewriteEngine On
# Always return 200 OK for OPTIONS method.
RewriteCond %{REQUEST_METHOD} OPTIONS
RewriteRule ^(.*)$ $1 [R=200,L]
</IfModule>

and use “always set” to overwrite cors headers in vhost section,

<VirtualHost *:80>
ServerName search.mysite.com
DocumentRoot /var/www/mysite/
<Directory /var/www/mysite>
AllowOverride All
Options -Indexes
</Directory>
<IfModule mod_headers.c>
SetEnvIf Origin "^(.*\.stockopedia\.com)$" ORIGIN_SUB_DOMAIN=$1
Header always set Access-Control-Allow-Origin "%{ORIGIN_SUB_DOMAIN}e" env=ORIGIN_SUB_DOMAIN
Header always set Access-Control-Allow-Headers "origin, x-requested-with, content-type"
Header always set Access-Control-Allow-Methods "GET, OPTIONS"
</IfModule>
</VirtualHost>

More details can be found at,

http://enable-cors.org/index.html

http://www.html5rocks.com/en/tutorials/cors/

Thanks.

Posted in Client-side Scripting. | Tagged , , | Leave a comment

Killing child process in shell script

Killing child process in shell script:

Many time we need to kill child process which are hanged or block for some reason. eg. FTP connection issue.

There are two approaches,

1) To create separate new parent for each child which will monitor and kill child process once timeout reached.

Create test.sh as follows,

#!/bin/bash
declare -a CMDs=("AAA" "BBB" "CCC" "DDD")
for CMD in ${CMDs[*]}; do
(sleep 10 & PID=$!; echo "Started $CMD => $PID"; sleep 5; echo "Killing $CMD => $PID"; kill $PID; echo "$CMD Completed.") &
done
exit;

and watch processes which are having name as ‘test’ in other terminal using following command.

watch -n1 'ps x -o "%p %r %c" | grep "test" '

Above script will create 4 new child processes and their parents. Each child process will run for 10sec. But once timeout of 5sec reach, thier respective parent processes will kill those childs.
So child won’t be able to complete execution(10sec).
Play around those timings(switch 10 and 5) to see another behaviour. In that case child will finish execution in 5sec before it reaches timeout of 10sec.

2) Let the current parent monitor and kill child process once timeout reached. This won’t create separate parent to monitor each child. Also you can manage all child processes properly within same parent.

Create test.sh as follows,

#!/bin/bash
declare -A CPIDs;
declare -a CMDs=("AAA" "BBB" "CCC" "DDD")
CMD_TIME=15;
for CMD in ${CMDs[*]}; do
(echo "Started..$CMD"; sleep $CMD_TIME; echo "$CMD Done";) &
CPIDs[$!]="$RN";
sleep 1;
done
GPID=$(ps -o pgid= $$);
CNT_TIME_OUT=10;
CNT=0;
while (true); do
declare -A TMP_CPIDs;
for PID in "${!CPIDs[@]}"; do
echo "Checking "${CPIDs[$PID]}"=>"$PID;
if ps -p $PID > /dev/null ; then
echo "-->"${CPIDs[$PID]}"=>"$PID" is running..";
TMP_CPIDs[$PID]=${CPIDs[$PID]};
else
echo "-->"${CPIDs[$PID]}"=>"$PID" is completed.";
fi
done
if [ ${#TMP_CPIDs[@]} == 0 ]; then
echo "All commands completed.";
break;
else
unset CPIDs;
declare -A CPIDs;
for PID in "${!TMP_CPIDs[@]}"; do
CPIDs[$PID]=${TMP_CPIDs[$PID]};
done
unset TMP_CPIDs;
if [ $CNT -gt $CNT_TIME_OUT ]; then
echo ${CPIDs[@]}"PIDs not reponding. Timeout reached $CNT sec. killing all childern with GPID $GPID..";
kill -- -$GPID;
fi
fi
CNT=$((CNT+1));
echo "waiting since $b secs..";
sleep 1;
done
exit;

and watch processes which are having name as ‘test’ in other terminal using following command.

watch -n1 'ps x -o "%p %r %c" | grep "test" '

Above script will create 4 new child processes. We are storing pids of all child processes and looping over them to check if they are finished their execution or still running.
Child process will execution till CMD_TIME time. But if CNT_TIME_OUT timeout reach , All children will get killed by parent process.
You can switch timing and play around with script to see behaviour.
One drawback of this approach is , it is using group id for killing all child tree. But parent process itself belong to same group so it will also get killed.

You may need to assign other group id to parent process if you don’t want parent to be killed.

Following is one more example which monitors php script and kills if it reaches timeout.

1) test.sh

#!/bin/bash
LOG='log.txt'
trap 'echo "LineNo.$LINENO" >> $LOG; exit 1;' ERR SIGINT SIGTERM;
CMD="php test.php;"
echo $CMD
eval $CMD &>> $LOG &
GPID=$(ps -o pgid= $$);
CPID=$!
echo "PIDs: $GPID - $CPID "
CNT=0;
CNT_TIME_OUT=10;
while (true); do
if ps -p $CPID > /dev/null ; then
echo "$CPID is running..";
if [ $CNT -gt $CNT_TIME_OUT ]; then
echo "Timeout reached $CNT_TIME_OUT sec. killing $GPID.. breaking..";
kill -- -$GPID;
fi
else
echo "$CPID is completed. Breaking..";
break;
fi
CNT=$((CNT+1));
echo "waiting since $b secs..";
sleep 1;
done
exit;

1) test.php

<?php
$i=0;
$l=300;
while($i<$l) {
#throw new \Exception("Testing");
$date = new DateTime();
$date->add(DateInterval::createFromDateString('yesterday'));
echo $date->format('Y-m-d H:i:s') . " => $i\n";
sleep(1);
$i++;
echo "End => $i\n";
}
die;
?>

Thanks.

Posted in Shell Script | Tagged | Leave a comment

Setup solr using zookeeper ensemble on ubnutu

solrcould-cluster-single-collection-zookeeper-ensemble

Setup Oracle Java:
Follow quick step given below to setup java latest version on your system,

java -version
tar -zxvf jdk-8u45-linux-x64.tar.gz
sudo mkdir -p /usr/lib/jvm/jdk1.8.0_45
sudo mv jdk1.8.0_45/* /usr/lib/jvm/jdk1.8.0_45/
sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.8.0_45/bin/java" 1
sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.8.0_45/bin/javac" 1
sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.8.0_45/bin/javaws" 1
sudo update-alternatives --set java /usr/lib/jvm/jdk1.8.0_45/bin/java
sudo update-alternatives --set javac /usr/lib/jvm/jdk1.8.0_45/bin/javac
sudo update-alternatives --set javaws /usr/lib/jvm/jdk1.8.0_45/bin/javaws
java -version
vim ~/.bashrc
export JAVA_HOME="/usr/lib/jvm/jdk1.8.0_45"
export PATH="$PATH:$JAVA_HOME/bin"
source ~/.bashrc
echo $JAVA_HOME
echo $PATH

For more detail kindly visit how-to-install-oracle-jdk-7-on-ubuntu-12-04

Setup Zookeeper Ensemble: (https://cwiki.apache.org/confluence/display/solr/Setting+Up+an+External+ZooKeeper+Ensemble)

Consider we have 2 servers (192.168.0.101, 192.168.0.111) and we are setting up 6 zookeeper nodes called znodes.
Follow quick step given below to setup Zookeeper Ensemble on your system,

wget 'http://mirror.symnds.com/software/Apache/zookeeper/stable/zookeeper-3.4.6.tar.gz'
tar -xvf zookeeper-3.4.6.tar.gz
sudo mkdir -p /usr/lib/zookeeper-3.4.6
sudo mv zookeeper-3.4.6/* /usr/lib/zookeeper-3.4.6/
cd /usr/lib/zookeeper-3.4.6/
sudo cp conf/zoo_sample.cfg conf/zoo_1.cfg
sudo vim conf/zoo_1.cfg

Add following configs,

tickTime=2000
dataDir=/var/lib/zookeeper/1/
clientPort=2181
initLimit=5
syncLimit=2
server.1=192.168.0.101:2888:3888
server.2=192.168.0.101:2889:3889
server.3=192.168.0.101:2890:3890
server.4=192.168.0.111:2888:3888
server.5=192.168.0.111:2889:3889
server.6=192.168.0.111:2890:3890

Repeat above setting for all servers.

Make data directories:(Repeat for all servers)

sudo mkdir -p /var/lib/zookeeper/1/
sudo sh -c 'echo "1" > /var/lib/zookeeper/1/myid'

Start servers:(Repeat for all servers)
sudo bin/zkServer.sh start zoo_1.cfg

Check Status:

bin/zkServer.sh status zoo_1.cfg
echo status | nc localhost 2187

Test Client:

bin/zkCli.sh -server 192.168.0.101:2181
[zk: 192.168.0.101:2181(CONNECTED) 1] ls /
[zk: 192.168.0.101:2181(CONNECTED) 2] get /configs/gettingstarted/solrconfig.xml
[zk: 192.168.0.101:2181(CONNECTED) 3] quit

Stop Servers:(Repeat for all servers)
sudo bin/zkServer.sh stop zoo_1.cfg

For more detail kindly visit Setup-zookeeper-ensemble-on-ubuntu

Setup Solr:

Consider we have 2 servers (192.168.0.101, 192.168.0.111) and we are setting up 2 solr instances in shard=1:replication:2 setup.
wget 'http://apache.mirrors.hoobly.com/lucene/solr/5.2.0/solr-5.2.0.tgz'
SolrCloud Standalone Setup using Embedded Zookeeper (Testing environment): (http://lucene.apache.org/solr/quickstart.html)
Please follow above link for demos.

SolrCloud Mode Setup using External Zookeeper Ensemble (Testing environment): (https://cwiki.apache.org/confluence/display/solr/Solr+Start+Script+Reference)
cd solr-5.2.0/tar -zxvf solr-5.2.0.tgz

Create Node:

mkdir -p example/cloud/node1/solr
cp server/solr/solr.xml example/cloud/node1/solr
mkdir -p example/cloud/node2/solr
cp server/solr/solr.xml example/cloud/node2/solr
bin/solr start -cloud -s example/cloud/node1/solr -h 192.168.0.101 -p 8983 -z 192.168.0.101:2181,192.168.0.101:2182,192.168.0.101:2183,192.168.0.111:2184,192.168.0.111:2185,192.168.0.111:2186
bin/solr start -cloud -s example/cloud/node1/solr -h 192.168.0.111 -p 8983 -z 192.168.0.101:2181,192.168.0.101:2182,192.168.0.101:2183,192.168.0.111:2184,192.168.0.111:2185,192.168.0.111:2186

Create Collection:
bin/solr create -c gettingstarted -d basic_configs -rf 2

Status:

bin/solr status
bin/solr healthcheck -c gettingstarted

Delete Collection:
bin/solr delete -c gettingstarted

Restart:

bin/solr restart -cloud -s example/cloud/node1/solr -h 192.168.0.101 -p 8983 -z 192.168.0.101:2181,192.168.0.101:2182,192.168.0.101:2183,192.168.0.111:2184,192.168.0.111:2185,192.168.0.111:2186
bin/solr restart -cloud -s example/cloud/node1/solr -h 192.168.0.111 -p 8983 -z 192.168.0.101:2181,192.168.0.101:2182,192.168.0.101:2183,192.168.0.111:2184,192.168.0.111:2185,192.168.0.111:2186

Stop Node:
bin/solr stop -all;

Clean all Testing files:
rm -Rf example/cloud/

SolrCloud as a Service using External Zookeeper Ensemble (Production environment): (https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production)

Run Installation script:

tar xzf solr-5.2.0.tgz solr-5.2.0/bin/install_solr_service.sh --strip-components=2
sudo bash ./install_solr_service.sh -help
sudo bash ./install_solr_service.sh solr-5.0.0.tgz -i /opt -d /var/solr -u solr -s solr -p 8983
OR
sudo bash ./install_solr_service.sh solr-5.2.0.tgz
id: solr: no such user
Creating new user: solr
Adding system user `solr' (UID 109) ...
Adding new group `solr' (GID 116) ...
Adding new user `solr' (UID 109) with group `solr' ...
Creating home directory `/home/solr' ...
Extracting solr-5.2.0.tgz to /opt
Creating /etc/init.d/solr script ...
Adding system startup for /etc/init.d/solr ...
Waiting to see Solr listening on port 8983 [/]
Started Solr server on port 8983 (pid=1704). Happy searching!
Service solr installed.
sudo service solr status
sudo service solr stop

Setup SolrCloud:

To run Solr in SorlCloud add following setting in environment specific include file (/var/solr/solr.in.sh),
ZK_HOST="192.168.0.101:2181,192.168.0.101:2182,192.168.0.101:2183,192.168.0.111:2184,192.168.0.111:2185,192.168.0.111:2186"

If you’re using a ZooKeeper instance that is shared by other systems, it’s recommended to isolate the SolrCloud znode tree using ZooKeeper’s chroot support. For instance, to ensure all znodes created by SolrCloud are stored under /solr, you can put /solr on the end of your ZK_HOST connection string, such as:
ZK_HOST="192.168.0.101:2181,192.168.0.101:2182,192.168.0.101:2183,192.168.0.111:2184,192.168.0.111:2185,192.168.0.111:2186/solr"

If using a chroot for the first time, you need to bootstrap the Solr znode tree in ZooKeeper by using the zkcli.sh script, such as:
/opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost 192.168.0.101:2181 -cmd bootstrap -solrhome /var/solr/data

If above script couldn’t be able to create ‘solr’ chroot because of 0 core (As we have not create any core yet) then create one by zookeeper client,

/usr/lib/zookeeper-3.4.6/bin/zkCli.sh -server 192.168.0.101:2181
[zk: 192.168.0.101:2181(CONNECTED) 2] create /solr solr

Note: Above fix is not specified anywhere in Solr Doc.
sudo service solr start

Upload a configuration directory: (https://cwiki.apache.org/confluence/display/solr/Command+Line+Utilities)
/opt/solr/server/scripts/cloud-scripts/zkcli.sh -zkhost 192.168.0.101:2181/solr -cmd upconfig -confname data_driven_schema_configs -confdir /opt/solr/server/solr/configsets/data_driven_schema_configs/conf

To delete wrong config use zookeeper client,

/usr/lib/zookeeper-3.4.6/bin/zkCli.sh -server 192.168.0.101:2181
[zk: 192.168.0.111:2184(CONNECTED) 8] rmr /configs

Create Collection:(https://cwiki.apache.org/confluence/display/solr/Collections+API)

http://host:port/solr/admin/collections?action=CREATE&name=gettingstarted&numShards=1&replicationFactor=1&maxShardsPerNode=1&collection.configName=data_driven_schema_configs

http://host:port/solr/admin/collections?action=DELETE&name=gettingstarted

For more detail kindly visit Install-solr-on-ubuntu

Important Download links,

Search Engine research:

https://www.elastic.co/products/elasticsearch

http://solr-vs-elasticsearch.com/

http://lucene.apache.org/solr/quickstart.html

https://cwiki.apache.org/confluence/display/solr/Apache+Solr+Reference+Guide

http://wiki.apache.org/solr/FrontPage

https://cwiki.apache.org/confluence/display/solr/Uploading+Structured+Data+Store+Data+with+the+Data+Import+Handler

Requirement:

http://lucene.apache.org/solr/5_1_0/SYSTEM_REQUIREMENTS.html

Downloads:

http://mirror.reverse.net/pub/apache/lucene/solr/5.1.0/solr-5.1.0.tgz

http://apache.mirrors.hoobly.com/lucene/solr/5.2.0/solr-5.2.0.tgz

http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.tar.gz

http://archive.apache.org/dist/lucene/solr/

http://mirror.symnds.com/software/Apache/zookeeper/stable/zookeeper-3.4.6.tar.gz

Setup:

https://cwiki.apache.org/confluence/display/solr/Taking+Solr+to+Production

http://zookeeper.apache.org/doc/r3.4.6/zookeeperStarted.html

PHP Client:

http://wiki.apache.org/solr/SolPHP

https://pecl.php.net/package/solr

http://php.net/manual/en/book.solr.php

Symfony Bundle:

https://packagist.org/packages/solarium/solarium

https://packagist.org/packages/nelmio/solarium-bundle

https://packagist.org/packages/reprovinci/solr-php-client

https://packagist.org/packages/floriansemm/solr-bundle

https://packagist.org/packages/internations/solr-utils

https://packagist.org/packages/internations/solr-query-component


Thank you.

Posted in Java, Solr, Uncategorized, Zookeeper | Tagged , , | Leave a comment

Setup ZooKeeper Ensemble on Ubuntu

Setup ZooKeeper Ensemble on Ubuntu:

Download Apache ZooKeeper:

The first step in setting up Apache ZooKeeper is, of course, to download the software. It’s available from http://zookeeper.apache.org/releases.html.

#wget http://mirror.symnds.com/software/Apache/zookeeper/stable/zookeeper-3.4.6.tar.gz
#tar -xvf zookeeper-3.4.6.tar.gz

Configure the instance:
Lets create one in conf/zoo1.cfg:

#sudo mkdir -p /usr/lib/zookeeper-3.4.6
#sudo mv zookeeper-3.4.6/* /usr/lib/zookeeper-3.4.6/
#cd /usr/lib/zookeeper-3.4.6/
#cp conf/zoo_sample.cfg conf/zoo1.cfg
#vim conf/zoo1.cfg

Add following settings,

tickTime=2000
dataDir=/var/lib/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=192.168.0.1:2888:3888
server.2=192.168.0.2:2888:3888
server.3=192.168.0.3:2888:3888

The parameters are as follows:
tickTime: Part of what ZooKeeper does is to determine which servers are up and running at any given time, and the minimum session time out is defined as two “ticks”. The tickTime parameter specifies, in miliseconds, how long each tick should be.
dataDir: This is the directory in which ZooKeeper will store data about the cluster. This directory should start out empty.
clientPort: This is the port on which Solr will access ZooKeeper.
initLimit: Amount of time, in ticks, to allow followers to connect and sync to a leader. In this case, you have 5 ticks, each of which is 2000 milliseconds long, so the server will wait as long as 10 seconds to connect and sync with the leader.
syncLimit: Amount of time, in ticks, to allow followers to sync with ZooKeeper. If followers fall too far behind a leader, they will be dropped.
server.X: These are the IDs and locations of all servers in the ensemble, the ports on which they communicate with each other. The server ID must additionally stored in the /myid file and be located in the dataDir of each ZooKeeper instance. The ID identifies each server, so in the case of this first instance, you would create the file /var/lib/zookeeper/1/myid with the content “1″.

Once this file is in place, you’re ready to start the ZooKeeper instance.

Then create /var/lib/zookeeper directory And create myid file, so each node can identify itself:

#sudo mkdir -p /var/lib/zookeeper
#echo "1" > /var/lib/zookeeper/myid

Where “1″ is the node number (so put “2″ for the next node and so on)
Do same for all nodes.

Standalone Setup:

You can also setup multiple instances on localhost. You just need to create separate data directory per instance for storing id and data and make all instances listen on different ports.

clientPort=2181
clientPort=2182
clientPort=2183

dataDir=/var/lib/zookeeper/1/
dataDir=/var/lib/zookeeper/2/
dataDir=/var/lib/zookeeper/3/

server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890

#echo "1" > /var/lib/zookeeper/1/myid
#echo "2" > /var/lib/zookeeper/2/myid
#echo "3" > /var/lib/zookeeper/3/myid

Once you have each node set up, you can start ZooKeeper by issuing on each node:

bin/zkServer.sh start zoo1.cfg
bin/zkServer.sh start zoo2.cfg
bin/zkServer.sh start zoo3.cfg

Check servers are running,
$bin/zkServer.sh status zoo1.cfg
$bin/zkServer.sh status zoo2.cfg
$bin/zkServer.sh status zoo3.cfg

$echo status | nc localhost 2181
$echo status | nc localhost 2182
$echo status | nc localhost 2183

Connect client,

/bin/zkCli.sh -server localhost:2181
[zk: localhost:2181(CONNECTED) 1] ls /
[zk: localhost:2181(CONNECTED) 2] ls /configs/
[zk: localhost:2181(CONNECTED) 3] ls /collections/
[zk: localhost:2181(CONNECTED) 4] get /configs/gettingstarted/solrconfig.xml
[zk: localhost:2181(CONNECTED) 5] quit

Stop them,

bin/zkServer.sh stop zoo1.cfg
bin/zkServer.sh stop zoo2.cfg
bin/zkServer.sh stop zoo3.cfg

Thanks you.

Posted in Zookeeper | Tagged | Leave a comment

How to install tor in ubuntu

tor-browser-download

How to install tor in ubuntu:

What is Tor?

Tor is free software and an open network that helps you defend against traffic analysis, a form of network surveillance that threatens personal freedom and privacy, confidential business activities and relationships, and state security.

Why Anonymity Matters?

Tor protects you by bouncing your communications around a distributed network of relays run by volunteers all around the world: it prevents somebody watching your Internet connection from learning what sites you visit, and it prevents the sites you visit from learning your physical location.

For more information visit torproject.

How to setup tor:

$wget https://www.torproject.org/dist/torbrowser/4.0.4/tor-browser-linux64-4.0.4_en-US.tar.xz
$tar xf tor-browser-linux64-4.0.4_en-US.tar.xz
$tor-browser_en-US/start-tor-browser

Thanks.

Posted in Tor | Tagged | Leave a comment

Setup zabbix on ubuntu

Setup zabbix on ubuntu:

Zabbix is the ultimate open source enterprise-level software designed for monitoring availability and performance of IT infrastructure components.
For more details visit following links,
www.zabbix.com
www.zabbix.org

Install Zabbix:

# wget http://repo.zabbix.com/zabbix/2.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_2.2-1+precise_all.deb
# sudo dpkg -i zabbix-release_2.2-1+precise_all.deb
# sudo apt-get update

Install Zabbix Component :

Zabbix server – a central process of Zabbix software that performs monitoring, interacts with Zabbix proxies and agents, calculates triggers, sends notifications; a central repository of data
# sudo apt-get install zabbix-server-mysql

Web frontend – the web interface provided with Zabbix
# sudo apt-get install zabbix-frontend-php

Zabbix agent – a process deployed on monitoring targets to actively monitor local resources and applications
# sudo apt-get install zabbix-agent

Zabbix proxy – a process that may collect data on behalf of Zabbix server, taking some processing load off of the server

Zabbix API – Zabbix API allows you to use the JSON RPC protocol to create, update and fetch Zabbix objects (like hosts, items, graphs and others) or perform any other custom tasks

Zabbix Setup :

PHP configuration for Zabbix frontend:

Apache configuration file for Zabbix frontend is located in /etc/apache2/conf.d/zabbix. Some PHP settings are already configured.

php_value max_execution_time 300
php_value memory_limit 128M
php_value post_max_size 16M
php_value upload_max_filesize 2M
php_value max_input_time 300
# php_value date.timezone Asia/Kolkata

It’s necessary to uncomment the “date.timezone” setting and set the correct timezone for you. After changing the configuration file restart the apache web server.
OR
sudo vim /etc/php5/apache2/php.ini
date.timezone = Asia/Kolkata
#sudo service apache2 restart

Follow installation steps here, http://localhost/zabbix/setup.php#

Zabbix configuration :

Appliance Zabbix setup has the following passwords and other configuration changes:

1: Passwords
System:
root:zabbix
zabbix:zabbix
Database:
root:zabbix
zabbix:zabbix
Zabbix frontend:
Admin:zabbix

If you change frontend password, do not forget to update password setting web monitoring (Configuration → Hosts, Web for host “Zabbix server”).

To change the database user password it has to be changed in the following locations:
MySQL;
zabbix_server.conf;
zabbix.conf.php.

2: File locations

Configuration files are placed in /etc.
Zabbix logfiles are placed in /var/log/zabbix.
Zabbix frontend is placed in /usr/share/zabbix.
Home directory for user zabbix is /etc/zabbix.

3: Changes to Zabbix configuration

Server name for Zabbix frontend set to “Zabbix 2.2 Appliance”;
Frontend timezone is set to Europe/Riga, Zabbix home (this can be modified in /etc/php5/apache2/php.ini);
Disabled triggers and web scenarios are shown by default to reduce confusion.

Thanks you.

Posted in Uncategorized, Zabbix | Tagged | Leave a comment

Setup vagrant on ubuntu

Setup vagrant on ubuntu:

Vagrant provides easy to configure, reproducible, and portable work environments built on top of industry-standard technology and controlled by a single consistent workflow to help maximize the productivity and flexibility of you and your team.
To achieve its magic, Vagrant stands on the shoulders of giants. Machines are provisioned on top of VirtualBox, VMware, AWS, or any other provider. Then, industry-standard provisioning tools such as shell scripts, Chef, or Puppet, can be used to automatically install and configure software on the machine.

More details can be found at vagrantup

Steps:

1) Install dependency :
VirtualBox is a general-purpose full virtualizer for x86 hardware, targeted at server, desktop and embedded use. (VirtualBox)

Add one of the following lines according to your distribution to your /etc/apt/sources.list:
deb http://download.virtualbox.org/virtualbox/debian precise contrib

#wget -q https://www.virtualbox.org/download/oracle_vbox.asc -O- | sudo apt-key add -
#sudo apt-get install virtualbox-4.3

2) Download and install vagrantup.

#wget https://dl.bintray.com/mitchellh/vagrant/vagrant_1.6.5_x86_64.deb
#sudo dpkg -i /home/hemant/Downloads/vagrant_1.6.5_x86_64.deb

3) Setup vagrant.
Create a root directory for your project and navigate in it:

#mkdir myproject
#cd myproject


Next, run the initialization command:

a) Pass box as parameter:
#vagrant init [box-name] [box-url]
If a first argument is given, it will prepopulate the config.vm.box setting in the created Vagrantfile. [ex. hashicorp/precise64]
If a second argument is given, it will prepopulate the config.vm.box_url setting in the created Vagrantfile. [ex. https://vagrantcloud.com/hashicorp/boxes/precise64]
b) Create a Vagrantfile and update default configuraion.

#vagrant init
#vim Vagrantfile

Modify file,
#config.vm.box = “precise64″

More Vagrantfile oprions,

#Install dependencies using provision,
config.vm.provision :shell, :path => "install.sh"
#Create a private network, which allows host-only access to the machine using a specific IP.
config.vm.network :private_network, ip: "192.168.1.1"
#Share an additional folder to the guest VM.
config.vm.synced_folder "/h_data", "/v_data", owner: "root", group: "root", :mount_options => ['dmode=777,fmode=777']

This will tell it to use this new box. More boxes can be discover here Vagrantcloud.
Save the file and exit. Now you can deploy the guest machine with the following command:

#vagrant up

This will bring up a VPS running Ubuntu 12.04 LTS 64Bit. To make use of it, you can easily SSH into it:

#vagrant ssh

Vagrant will share the project root folder from the host machine (the one containing the Vagrantfile) with a folder on the guest machine, /vagrant.

You can exit and go back to the host with the following command:

#exit

To stop and remove the guest machine and all traces of it,

#vagrant destroy

Use reload command is usually required for changes made in the Vagrantfile to take effect. After making any modifications to the Vagrantfile

#vagrant reload

The configured provisioners will not run again, by default. You can force the provisioners to re-run by specifying the –provision flag.

To add more boxes for other projects use following command adds ‘hashicorp/precise64′ box to Vagrant.

#vagrant box add hashicorp/precise64

Thanks you.

Posted in Virtualization | Tagged , | Leave a comment

Ubuntu fix Shellshock bug

Ubuntu fix Shellshock Bug :

Test if your system is vulnerable,

$ env X="() { :;}; echo shellshock" `which bash` -c "echo completed"
shellshock
completed

If you got “shellshock” in output then it is vulnerable.

Bash has functions, though in a somewhat limited implementation, and it is possible to put these Bash functions into environment variables. This flaw is triggered when extra code is added to the end of these function definitions (inside the enivronment variable).

Download package as per your OS version from fix
For 12.04 64bit LTS,

#wget https://launchpad.net/~ubuntu-security/+archive/ubuntu/ppa/+build/6400987/+files/bash_4.2-2ubuntu2.2_amd64.deb
#dpkg -i /home/hemant/Downloads/bash_4.2-2ubuntu2.2_amd64.deb

The patch used to fix this flaw, ensures that no code is allowed after the end of a Bash function.
So if you run the above example with the patched version of Bash, you should get an output similar to:

$ env X="() { :;}; echo shellshock" `which bash` -c "echo completed"
/bin/bash: warning: X: ignoring function definition attempt
/bin/bash: error importing function definition for `X'
completed

Now, you are safe from Shockshell bug.

For details can be found at,

Redhat

NVD

Thank you.

Posted in Uncategorized | Leave a comment

PHP multiprocessing using fork

PHP multiprocessing using fork:

Process Control support in PHP implements the Unix style of process creation, program execution, signal handling and process termination.

Find more details at PHP fork

The pcntl_fork() function creates a child process that differs from the parent process only in its PID and PPID.
On success, the PID of the child process is returned in the parent’s thread of execution, and a 0 is returned in the child’s thread of execution.
On failure, a -1 will be returned in the parent’s context, no child process will be created, and a PHP error is raised.
The pcntl_wait($status) function protect against zombie children.

PHP fork is useful when we want process multiple task parallely.

Let’s assume we have following task to perform.
1) updateCache : Time 3s.
1) sentEmail : Time 2s.
1) logEntry : Time 1s.
1) addDBEntry : Time 5s.

If we perform above task in serially the it will take 10 sec(5+3+2+1).
But, if we use parallel processing then it will reduce to 5 sec(largest time among them) using wait to protect against zombie children without returning immediately.

You can also return parent process and make child processes run in background as zombie.

$time_start = microtime(true);

function executeTask($types){

foreach($types as $type){
switch($type){
case 'updateCache':
sleep(3);
echo "updateCache \n";
break;

case 'sentEmail':
sleep(2);
echo "sentEmail \n";
break;

case 'logEntry':
sleep(1);
echo "logEntry \n";
break;

case 'addDBEntry':
sleep(5);
echo "addDBEntry \n";
break;
default:
break;
}
}
}

$useFork = true;

if($useFork){
$pid1 = pcntl_fork();
if ($pid1 == -1) {
$types = array("updateCache","sentEmail","logEntry","addDBEntry");
executeTask($types);
} else if ($pid1) {
$pid2 = pcntl_fork();
if ($pid2 == -1) {
$types = array("updateCache","sentEmail","logEntry");
executeTask($types);
} else if ($pid2) {
$types = array("updateCache");
executeTask($types);
} else {
$types = array("sentEmail","logEntry");
executeTask($types);
}
pcntl_wait($status);

} else {
$types = array("addDBEntry");
executeTask($types);
}

pcntl_wait($status);
if($pid1 && $pid2){
$time_end = microtime(true);
$time = $time_end - $time_start;
echo "Total time: $time seconds\n";
}
} else {
$types = array("updateCache","sentEmail","logEntry","addDBEntry");
executeTask($types);
$time_end = microtime(true);
$time = $time_end - $time_start;
echo "Total time: $time seconds\n";

}

Thank you.

Posted in Php | Leave a comment