Web Crawlers Search engines and User-Agents

Web Crawlers

Web crawlers, also known as web spiders or internet bots, are programs that browse the web in an automated manner for the purpose of indexing content. Crawlers can look at all sorts of data such as content, links on a page, broken links, sitemaps, and HTML code validation.

Search engines like Google, Bing, and Yahoo use crawlers to properly index downloaded pages so that users can find them them faster and more efficiently when they are searching. Without crawlers there would be nothing to tell them that your website has new and fresh content. Sitemaps also can play a part in that process. So web crawlers, for the most part, are a good thing. However there are also issues sometimes when it comes to scheduling and load as a crawler might be constantly polling your site. And this is where a robots.txt file comes into play. This file can help control the crawl traffic and ensure that it doesn’t overwhelm your server.

Web crawlers identify themselves to a web server by using the User-agent field in an HTTP request, and each crawler has their own unique identifier. Most of the time you will need to examine your web server referrer logs to view web crawler traffic.

Robots.txt
By placing a robots.txt file at the root of your web server you can define rules for web crawlers such as allow or disallow that they must follow. You can apply generic rules which apply to all bots or get more granular and specify their specific User-agent string.

Learn more about the Top Search engine Bots
There are hundreds of web crawlers and bots scouring the internet but below is a list of popular web crawlers and bots that we have  been collected based on ones that we see on a regular basis within our web server logs.

Googlebot: Googlebot is Google’s web crawling bot (sometimes also called a “spider”). Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site. Googlebot’s crawl process  begins with a list of webpage URLs, generated from previous crawl processes and augmented with Sitemap data provided by webmasters. As Googlebot visits each of these websites it detects links (SRC and HREF) on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.

"Google+"Another one you might see popup is Google+. When a user shares a URL on Google+ or an app writes an app activity, Google+ attempts to fetch the content and create a snippet to provide a summary of the linked content. This service is different than the Googlebot that crawls and indexes your site. These requests do not honor robots.txt or other crawl mechanisms because this is a user-initiated request.

"Most Popular And Best Internet Search Engines in the World"

"Baiduspider " Baiduspider is a robot of Baidu Chinese search engine. Baidu (Chinese: 百度; pinyin: Bǎidù) is the leading Chinese search engine for websites, audio files, and images.

"MSN Bot/Bingbot" This is a web-crawling robot (type of Internet bot), deployed by Microsoft to supply Bing (search engine). It collects documents from the web to build a searchable index for the Bing (search engine).
   

"Slurp Bot"Yahoo Search results come from the Yahoo web crawler Slurp and Bing’s web crawler, as a lot of Yahoo is now powered by Bing. Sites should allow Yahoo Slurp access in order to appear in Yahoo Mobile Search results.dditionally, Slurp does the following:

Collects content from partner sites for inclusion within sites like Yahoo News, Yahoo Finance and Yahoo Sports.
Accesses pages from sites across the Web to confirm accuracy and improve Yahoo’s personalized content for our users.
"Yandex Bot" Yandex bot is Yandex’s search engine’s crawler. Yandex is a Russian Internet company which operates the largest search engine in Russia with about 60% market share in that country. Yandex ranked as the fifth largest search engine worldwide with more than 150 million searches per day as of April 2012 and more than 25.5 million visitors.

"Soso Spider " Soso.com is a Chinese search engine owned by Tencent Holdings Limited, which is well known for its other creation QQ. Soso.com is ranked as the 36th most visited website in the world and the 13th most visited website in China, according to Alexa Internet. On an average, Soso.com gets 21,064,490 page views everyday.

"DuckDuckBot"DuckDuckBot is the Web crawler for DuckDuckGo, a search engine that has become quite popular lately as it is known for privacy and not tracking you. It now handles over 12 million queries per day. DuckDuckGo gets its results from over four hundred sources. These include hundreds of vertical sources delivering niche Instant Answers, DuckDuckBot (their crawler) and crowd-sourced sites (Wikipedia). They also have more traditional links in the search results, which they source from Yahoo!, Yandex and Bing.

"Baiduspider"Baiduspider is the official name of the Chinese Baidu search engine’s web crawling spider. It crawls web pages and returns updates to the Baidu index. Baidu is the leading Chinese search engine that takes an 80% share of the overall search engine market of China Mainland.

"Sogou Spider"Sogou Spider is the web crawler for Sogou.com, a leading Chinese search engine that was launched in 2004. it has a rank of 103 in Alexa’s internet rankings. Note: The Sogou web spider does not respect the robots.txt internet standard, and is therefore banned from many web sites because of excessive crawling.
"Exabot"Exabot is a web crawler for Exalead, which is a search engine based out of France. It was founded in 2000 and now has more than 16 billion pages currently indexed.

"Facebook External Hit"Facebook allows its users to send links to interesting web content to other Facebook users. Part of how this works on the Facebook system involves the temporary display of certain images or details related to the web content, such as the title of the webpage or the embed tag of a video.
"Alexa Crawler"Ia_archiver is the web crawler for Amazon’s Alexa internet rankings. As you probably know they collect information to show rankings for both local and international sites.
"Google Feedfetcher" Used by Google to grab RSS or Atom feeds when users choose to add them to their Google homepage or Google Reader. Feedfetcher collects and periodically refreshes these user-initiated feeds, but does not index them in Blog Search or Google’s other search services (feeds appear in the search results only if they’ve been crawled by Googlebot).

Read More →

Best Online Free Conference Call Services

Free Conference Call is an excellent  and important online service. We do aconference call for many purposes. We can connect with all of our friends at the same time in a conference call. A conference call is a telephone call in which someone talks to more than one people at the same time. We can use this technique in  online and we can make the free conference call. An online free conference call is an excellent service. There are a lot of applications/software that enables you to make free conference calls/video conferencing. I will give you a list of best free conference call services.

Skype

Skype is the best and most popular free conference call service. It is an application that allows the users video calls and video chat. Skype is a free VOIP. Skype is really easy to use, just download onto your computer and invites other to join. You can easily make a conference call using Skype. Skype is available for Windows, Android, Mac, Linux, iOS, Blackberry, Tablets and more. It was developed by Skype Technologies(Microsoft Corporation) in 2003. This free conference call service is available in 38 languages. 

CLICK HERE

FreeConferencePro

FreeConferencePro is one of the best free conference call services. It is really popular among the conference call users. You can easily make a conference call using this service. FreeConferencePro will record a custom greeting for you for free besides conference call.

CLICK HERE

Google Voice

Google Voice is an excellent free conference call service. It is a telephony service and provides call forwarding and voicemail services. It also provides  international call termination services for Google Account customers. Google Voice was developed by Google in 2009. you can easily make a free conference call using Google Voice.

CLICK HERE

Go ToMeeting

Go ToMeeting is one of the most popular free conference call services. It is an online meeting, desktop sharing and video conferencing software. It enables the users to meet with other computer users, customers, clients and more via internet. It was developed by Citrix. It is really easy to use.

CLICK HERE

UberConference

UberConference is an excellent, cloud-based congerencing service. It was developed by Switch Communications, Inc. in 2012. It is a California-based Telephony service. UberConference provides you a visual way to who is talking with you. 

   

CLICK HERE

SightSpeed

SightSpeed is one of the best video and voice calling service for Mac and PC. It is an excellent free conference call service. SightSpeed provides VoIP and instant messaging software for Windows and Apple Macintos operating systems. It was developed by SightSpeed Inc. in 2007. 

CLICK HERE

Free Conference Calling

Free Conference Calling is a free conference call service. It provides you free conference calling and other facilities so that you can make a conference call without any hesitation. You can create calls that consist  of  up to a 1,000 people and you get all the details about your conference call on the Dashboard. 

CLICK HERE

Dimdim

Dimdim is a free conference call service. It is easy, open, online meetings and weinares. No installation required to use this service. It is easy to use. It provides one-click audio, video and screen sharing and instant meeting capabilities. 

CLICK HERE

Tokbox

Tokbox is a free conference call service. Tokbox offers free group video conferencing.  You can connect face to face with 200 people. You need not any download, just click and use.

CLICK HERE

MegaMeeting

MegaMeeting is a web-based conferencing service. It provides web and video conferencing software for online meeting. This software is compatible with Windows, Mac, Linux. MegaMeeting was founded by Dan Richmond and Scott Goldman in 2003. It has 30 days free conference call service.

CLICK HERE 

Read More →

Best Screenshot Apps for Android

The two-button combo for screenshots is easy but what if our smartphone has a fragile lock or volume button, we wouldn’t want the buttons to break, would we? Along with that, we sometimes need editing tools for doodling, arrows, boxes etc.. to point out something in screenshots before we can post them on social networks. There are a number of screenshot apps on the Google Play Store that not only allow you to take screenshots with ease, they also bring some cool additional features.
We tried many apps on the Play Store and here are the best screenshot apps for Android (in no particular order):
Note: Most of the screenshot apps on Android come with various triggers to take screenshots. Not every trigger will work on all smartphones due to access or hardware limitations. 

Best Screenshot Apps for Android

Screenshot Easy

Screenshot Easy is a very popular Android app for taking screenshots with ease. The app has a very simple UI, which works well with Android’s Material Design UI guidelines. There are various triggers in the app, which lets you take a screenshot. You can choose to take a screenshot through an overlay button, which keeps floating or using a button in the notification center or by shaking your device. The app also lets you set your hardware camera button to take screenshots and that’s if your smartphone has a hardware camera button. If the app is running in the background and you take a screenshot the conventional way, it detects that and opens the screenshot in the app for your editing needs.

 

When it comes to the editing features, it lets you crop your screenshots, ZIP it or print it. You can also change existing colors in a screenshot to the one you like. There are options to show date & time on screenshots and save the images in PNG or JPG format. Other features of the app include themes, support for locale & tasker plugins. Moreover, the app is available for free on the Play Store.
Install: (Free)
Compatibility: Android 4.0 and up.

Screenshot Ultimate

Screenshot Ultimate is not the best looking Android app out there but if you want a ton of features in a screenshot app, Screenshot Ultimate won’t disappoint you. It has the most number of triggers to take a screenshot and you might find some of them to be really convoluted. You can take screenshots through an overlay button, notification bar button, shaking your device, power connect/disconnect and more. There are advanced triggers like audio trigger (blow into your device’s mic to take screenshot), interval (set time period to automatically take screenshots), cronjob (set when to take a screenshot in a specific app), proximity (put finger on proximity sensor to take screenshots), intent (tasker & locale support) and more.



Along with the various triggers, the app lets you make adjustments to your screenshots, change formats and more. There are also editing features like draw, crop, add text, rotate, mirror, effects and more. It also lets you share images directly to Imgur. The good news is Screenshot Ultimate is available for free on the Play Store.
Install: (Free)
Compatibility: Android 2.1 and up.

Screenshot Snap

While the Screenshot Snap app doesn’t really bring any intuitive methods to take a screenshot, it makes up with its diverse editing features. The app detects a screenshot being taken and loads it up on the app. You can then edit the screenshot on the go and share it quickly. You can doodle on an image with the pencil or paintbrush tool and you can also insert arrows, boxes, text and more.



The app brings you the option to save images in PNG or JPG, increase or decrease image quality. Screenshot Snap is available in a free version but it is limited when it comes to features and the ads can be a little annoying. The paid version of the app brings some cool stamps, blur tool, cloud uploading and removes ads.
Install: (Free, Paid $0.99)
Compatibility: Android 4.0 and up.

Perfect Screen Shot

Perfect Screen Shot is another cool screenshot app, which stands out due to its unique features. The app feels smooth in operation and it embodies Android’s Material Design UI, if that’s a deal breaker for you. While the app does not take screenshots for you, it detects when one is taken and opens the app with the screenshot right in front. It lets you frame your screenshots inside a device’s frame and you can edit the colors, background and more. There are various options to edit the frames and you can add glare, shadows & blur to the images. You can get frames of various devices ranging from Nexus devices such as Nexus 6P, Nexus 5X to Galaxy S6 Edge, One M9, Moto G and more.



The free “Classic” version of the app is pretty limited when it comes to the number of frames and editing features. The paid “Ultra” version brings you 3D frames, various editing options like background color, custom background images, optional blur and more.  You can also choose to edit your screenshots without a frame, if frames are not really your thing. The app also lets you import screenshots from any cloud storage services that you use like Google Drive, OneDrive, Dropbox etc.. To sum it up, Perfect Screen Shot is a cooler way to share screenshots and those screenshots certainly do look better inside a device frame.
Install: (Free Classic, Paid Ultra $1.99)
Compatibility: Android 4.0.3 and up.

Capture Screenshot

Capture Screenshot is very similar to Screenshot Easy but Capture Screenshot is a great screenshot app in its own right. The app is fast and is designed beautifully, so full marks to the developers on that front. It brings you various triggers like overlay button, notification shade button and shake to capture. Once you click the “Start Service” button, the app starts running in the background and detects whatever screenshots you take.



Once a screenshot is taken, Capture Screenshot lets you crop, draw, rotate, invert, add text and more. It also brings a number of filters like Boost, Grayscale, Gamma, Hue, Vignette etc.. There’s ability to save screenshots in PNG or JPG image formats. Along with that, the app brings you themes and you can choose the color you like for the app to give it a personalized look. While the free version of Capture Screenshot packs in most of the features, the “Pro” version removes ads and brings more filters.
Install (Free, Pro version $3)
Compatibility: Android 2.3 and up.

Read More →

Websites to Download Creative Commons Music For Free

If you’ve wondered where many of those YouTube videos get their background music from, you’ve come to the right place. Music with Creative Commons licenses are music compositions written, produced and shared by people who do not charge anything when you use their music for your own use, commercially or non-commercially.

Basically, they make the music and keep ownership and copyright of it and you get the permission to use it for your videos, film or small multimedia productions, as long as you credit the creator properly.

There are many sites out there that house music with Creative Commons licenses, sites where you can download music for free (and legally). These sites make it very easy for you to use, all you have to do is pick an audio clip you like and save it as an MP3, and there you have it, your free music download.

Jamendo

Jamendo has more than 350,000 audio files on their database, which includes music in a few different languages and from different countries. Its advanced search will help you narrow down what you’re looking for on their large database.

There is also a ‘Radio’ option where you can listen to music like on a radio channel based on genre. By registering and logging in, you can save personal favourites to your account.

Freesound

Freesound is a collaborative database for Creative Commons Licensed sounds. This database here is filled with sound effect such as ambient noises, synthesized sounds and sounds produced by musical instruments.

Browse, download and share sounds on Freesound; you can also upload your own audio clips to their database after registering for an account on the website.

ccMixter

Music on this site can be downloaded and shared with anyone, anywhere, anytime. Clips here focus on instrumental samples that are mainly used for remixes on DJ music. There is a wide variety of samples to choose from, which can be sorted by tags that describe the audio clips.

Kompoz

Kompoz is an online community that prides in collaboration between artists from all over the world. Artists post their clips on the website where anyone can download it and use it for their own productions. They can also add their own instruments or creative mix to the original clip and upload it on the website again.

As more people collaborate on the original clip and the clips after that, it will eventually become a creative collaboration of multiple musicians.

SoundCloud

SoundCloud is a very popular website where famous artists share their new tracks to the world. There is a section under Creative Commons license where you can download other people’s tracks. You can stream the available clips and it takes just a simple click to get the MP3 file without any registration.

Note that some tracks however require you to ‘Like’ their Facebook page before you can download their clips.

BeatPick

On BeatPick, you have a music player kind of interface where you can listen to any song on a list. To download any song from the website, you’ll have to register for an acoount. Most of the clips are instrumental, with very little vocal input.

The clips also come with descriptions of what it sounds like, which can be useful to match to whatever the theme your clip is.

Restorm

This is a website mostly used by indie artists who want a platform to sell their music to the public and have their works licensed under their name. You too can do the same, or download some of their free selection of music just by creating an account.

The full songs available on the website can be streamed to help you decide if you want to download it. Some of the artists also have songs in their native language not just English.

Musopen

Musopen is a website where you can find free music from individual instruments in an orchestra: violins, cellos, violas, etc as well as a combination of instruments. It caters to the genre of quieter music like classical music or jazz.. Best of all, you can also download sheet music.

You can also browse music by sorting it by composer, instrument and even the period of time the music clip resembles.

Vimeo

You might recognize Vimeo as a place only for videos, but it also offers free music which you can use as long as you give credit to the original artist. After clicking download, your download will start and a pop-up window will appear with the details of the clip’s title, artist and website.

For example, when using a music for a YouTube video, you would use those details and place it at the credits of your video or video description.

ArtistServer

ArtistServer has over 11,000 MP3 downloads by more than 10,000 artists and you can be one of those artists too when you sign up with them. They have clips of many genres for you to choose from which you can download for free.

It is also possible to preview the entire clip before making your selection and downloading it.

Audiofarm

Audiofarm has a wide selection of genres to choose from, uploaded by ordinary people who want to share their music. You can easily download any song you like by clicking the download link – no registration needed.

The full song can even be played while browsing for one that fits your needs. By registering, you can upload music for other people to download.

iBeat

If you’re a DJ or anyone else looking for some free beats and loops, iBeat has a wide database of clips which are free under Creative Commons licensing. The beats available come in a variety of genres such as rock beats, hip hop beats and even acoustic or electronic beats. No registration required.

CCTrax

CCTrax is a directory of free music. It has a database of Creative Commons license music for music lovers. Plenty of awesome sound track albums are available for free download here, like Alexander Saykov, Baumfreun, Stiver and more.

JewelBeat

On JewelBeat, you search through free background music and sound effects with keywords. These free background music clips can be used for whatever video production you are making. The audio clips here are mostly instrumental and consist of short loops. Besides the free selection, there is also a $0.99 selection that you can choose from.

Audionautix

Clips found on Audionautix are released under Creative Commons license 3.0 – you can use the audio clips available even for commercial purposes so long as you credit the Audionautix website. Clips are named to what the artist feels it should resemble; you can preview the clip by clicking it and if you like it, right click the clip and select ‘save target/link as’ to save it to your desktop.

FMA

FMA stands for Free Music Archive which has a wide selectionof high-quality, legal audio downloads. It works like a music station where you browse for clips and if you like the clip, you can add it to your music player. You also can opt to download the full clip. You can use the music player and download the clips without registering for an account.

Purple Planet

Purple planet is the creation of two people, Chris Martyn and Geoff Harvey, who let you use their music for free. You can use any of their audio clips on online videos so long as you link to their website at the description of the video. Their style of music falls under effects that accompany a horror, dramatic or mysterious scene.

Incompetech

Incompetech hosts a wide variety of royalty free music. You can browse their selection by genre or feel. "Feel" would include things like: Action, Relaxed, Intense or Humorous; you can select a combination of ‘Feels’ and it will check its database for selection matches. Clips can be played on the website and instantly downloaded as well.

Bump Foot

Bump Foot originates from Japan and focuses mostly on techno, trance beats and electronic dance music. You can download the files and use it for your own non-commercial work as long as you give attribution to them or the featured artist.

There is no way to stream music as there are only download links avalable on the site. You don’t need to register for an account to start downloading.

More

1. CASH Music – CASH Music is a non-profit organization that hosts digital tools for musicians and labels to share their music. The digital tool allows for uploads and downloads of songs and sharing is made easy.
2. Josh Woodward – This work of a single person, this site features more than 180 songs that are free for download, sharing and use. Do give this generous man attribution if you use his clips in your productions.
3. PacDV – This is a small database of free audio clips organised by mood or emotion such as: Warm, Loving, Serious or Lonely.
4. SampleSwap – SampleSwap is also a forum for other musicians. When a clip is uploaded, the floor is upon for discussion about the clip, as well as a free download.
5. DanoSongs – This is also a one-man wonder site. You get to use his wide variety of self-made clips, arranged by genre such as soundtrack, pop/rock band and even techno/electric.
6. Public Domain 4U – This website has a wide selection of genres of free music you can download. The genre leads to another website which houses the album and description of the album. You can download your tracks there without registration.
7. Orphan Songs – This website is created by Carl-Otto Johansson who is a musician who writes and records his own music. He uploads his two albums and a few tracks licensed under Creative Commons for public use.
8. Sonnyboo – To use songs from this website, you have to credit the owner and writer of the songs, Peter John Ross. The song selection he has ranges from Jazz to

Read More →