Category Archives: Link Building

Search Engine Spider and User Agent Identification with “Ultimate User Agent Blacklist”

Search Engine Spider and User Agent Identification with “Ultimate User Agent Blacklist”

 

A user-agent is software (a software agent) that is acting on behalf of a user.
In many cases, a user-agent acts as a client in a network protocol used in communications within a client–server distributed computing system.

 

For more information view wiki
http://en.wikipedia.org/wiki/User_agent

 

Automated Agents is called as Bots.
http://user-agents.org has a complete list of all latest spiders/bots/user agents.
There are some more and those are anonymous (not known and have very different names).

 

If you develop a website and want to make the site accessible by some specific user agent or bots of a country, then you can update the information in your root .htaccess file.

 

As, Bot blocking blacklists are useless in some way as some rogue spiders just generate random user-agent strings so we will never have them in our list to start with, but We have tried list out as much as we can in the below zip file.

 

Ultimate User Agent Blacklist

 

What you have to done is, unzip the file and paste the code in your root .htaccess file, it will protect your website from unwanted crawling/indexing by anonymous bots.

 

If your website uses WordPress open-source, then can also use “Better WP Security” plug-in.
Just need to download and install the plug-in and go to “Better WP Security – Ban Users” Tab and enable “Enable Default Banned List”, you can also update the list according to your needs.

 

But please be careful before doing same, as it may affect your website’s core files and plug-in, So before doing this backup your website’s files and database.

 

Integrate Twitter Widget to add to your website

The twitter updates are always short – under 140 characters each. Plus, one can post updates and follow her friends using the Twitter website, software on his browser, a mobile phone or instant messages. People aren’t tied to one device. This makes the twitter so popular.

 

It has grown far beyond its microblogging roots to become a fabulous tool for social messaging.

Many people use Twitter clients to update their status and read tweets, but there are also a host of useful Twitter widgets that allow you to check your tweets from your blog/website or even let people re-tweet your blog entries.

 

Display your Twitter updates on your website like this.

twitter-widget

This is one of Twitter widgets that will let you take your status updates and put them up anywhere that allows custom widgets. The great thing about the Twitter Profile Widget is that you can put your tweets on a loop.

 

Twitter provide very rich widgets, where you can easily manage it’s width, height, background colour, text colour, links, number of tweets, loop scroll etc.

 

You can produce the code by follow the below link
https://twitter.com/about/resources/widgets/widget_profile

 

Googlebot & Site Crawl

Advice Googlebot (Google) To Crawl Your Site

 

Googlebot is a Google’s web crawling bot or spider. This collects data from the web pages to build a searchable index for the Google search engine. Crawling is simply a process by which Googlebot visits new and updated pages, It uses an algorithmic programs determine which sites to crawl, how often, and how many pages to fetch from each site?

 

As Googlebot visits website it detects links (src and href) on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

 

If a webmaster wishes to control the information on their site available to a Googlebot,they can do so with the appropriate directives in a robots.txt file, or by adding the meta tag

 

<meta name=”Googlebot” content=”nofollow” />

 

to the web page.

 

Once you’ve created your robots.txt file, there may be a small delay before Googlebot discovers your changes.

 

Googlebot discovers pages by visiting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to other known pages on the web in order to be crawled and indexed or manually submitted by the webmaster.

 

Facebook Twitter and Google plus share and Like Button

Now a days in very busy life and in internet edge, social website play a very important role in friendship and social work and sharing views and idea, wishing Birthday and Anniversary.

 

Social websites now provide various API that can be implemented in websites(personal or commercial), It will provide live-streaming, sharing links etc.

 

Facebook twitter and Google plus along with LinkedIn provide us like/share button API, by implementing it, it will show a button on your webpage, the Like button lets a user share the page content with friends on social websites. When the user clicks the Like button on site, a related story appears in the user’s friends’ News Feed with a link back to website (from where a user clicked button).

 

This means when a user clicks a Like button on your page, a connection is made between webpage page and the user.

 

Facebook Like

There are two ways to Facebook Like button implementations.One is XFBML and second one is Iframe.

The XFBML dynamically re-sizes button’s height according to whether there are profile pictures to display,gives the ability to know in real time when a user clicks the Like button, and it gives the user the ability to add an optional comment to the like.

 
XFBML
<div id="fb-root"></div>
<script>(function(d, s, id) {
var js, fjs = d.getElementsByTagName(s)[0];
if (d.getElementById(id)) {return;}
js = d.createElement(s); js.id = id;
js.src = "//connect.facebook.net/en_US/all.js#xfbml=1";
fjs.parentNode.insertBefore(js, fjs);
}(document, 'script', 'facebook-jssdk'));</script>
<div class="fb-like" data-send="true" data-width="450" data-show-faces="true"></div>
 
IFRAME
<iframe src="//www.facebook.com/plugins/like.php?href=<?=urlencode($yourpageurl)?>&amp;send=false&amp;layout=button_count&amp;width=450&amp;show_faces=false&amp;action=like&amp;colorscheme=light&amp;font=arial&amp;height=21" scrolling="no" frameborder="0" style="border:none; overflow:hidden; width:450px; height:21px;" allowTransparency="true"></iframe>

 

Twitter Share

<a href="https://twitter.com/share" data-url="<?=$yourpageurl?>" data-count="horizontal">Tweet</a>
<script type="text/javascript" src="//platform.twitter.com/widgets.js"></script>

 

Google plus Share

<script type="text/javascript" src="https://apis.google.com/js/plusone.js"></script>
<div class="g-plusone" data-size="medium"></div>

These are the basic code, There are so many options available on these button, a compete implementation guide available here.