Webmonkey » search

Google Discontinues Site-Blocking Service

Scott Gilbertson — Mon, 25 Mar 2013 15:35:12 +0000

Image: THOR/Flickr

The hits just keep getting killed off. Google is shutting down yet another service — the company’s domain blocking tool, which allowed logged-in users to block unwanted domains from Google’s search results.

Google’s site-blocking tool was originally aimed at “content farm spam,” but Google hasn’t done much with it of late, and it even stopped working for a while, despite being available via a link from your profile.

Now the service is officially gone, replaced by a Chrome add-on that does nearly the same thing. Unfortunately that means the ability to ban sites from Google’s search results is now limited to those using Google’s Chrome web browser. For more on the Chrome add-on see our earlier review.

The bad news about the Chrome extension is that it’s client-side filtering, not server-side. That means that if Google returns results from domains you’ve blocked those results are simply hidden (sometimes there’s even a brief flash of the blocked results).

That means you’ll end up with fewer search results than you would with the server-side solution, which filtered out your blocked domains before the results were sent. For example, if there are ten results on the first page and three are from domains you’ve blocked, using the add-on method you’ll only see seven results, whereas the server-side method would have fetched the next three results to show a total of ten.

If you used the account-based version of the blocking tool, you can head over to your account and grab the list of sites you had blocked. Just add those sites to the Chrome extension and you’ll be back up and running in no time, with not an Experts-Exchange, Quora or W3Schools link to be seen (or whatever you consider search results spam).

Home Page Photo: Carlos Luna / Flickr

You Suck at Search

Scott Gilbertson — Tue, 19 Mar 2013 13:18:21 +0000

Image: Screenshot/Webmonkey

Even if you’re pretty good at searching, the majority of your website’s users are probably not. In fact, user experience expert Jakob Nielsen thinks most people are so bad at searching that site-specific search engines would do better to return navigation elements rather than actual search results.

Nielson’s research reveals that while more people reach for the search box to find what they’re after on a site, few of them “know how to use it.” The normally more prosaic Nielson writes:

It would certainly be nice if schools would get better at teaching kids how to search. But I don’t hold out much hope, because most people have the literary skills of an anteater (I was going to say, “a chimpanzee,” but these animals are too smart for my metaphor). Having new and varied vocabulary words spring from their foreheads wasn’t a survival skill for ice age hunters, so most people today can’t think up good queries without help.

Presumably Nielsen means literacy skills, not literary skills. That’s a pretty harsh critique, but if you’ve ever watched a less web-savvy friend or family member search for something you might be able to relate.

So how do you design your site’s search tool to help these “mediocre searchers” as Nielsen calls them?

Nielsen is critical of instant search suggestions, currently a popular way to help people using search tools. He claims that, while sometimes helpful, auto-complete tools can also be limiting because “users often view the drop-down as a mini-SERP and assume that it lists everything the site carries.”

The better way to do search according to Nielsen is to simply return product categories. The example in his report cites Costco, which, when searching for “television” will return all of its TV product categories rather than actual individual televisions. The product category links help users refine their choice and get to the televisions they actually want without having to wade through as many individual results.

It’s important to note that Nielsen is only advocating this sort of redirecting when the search term is “unambiguous and exactly matches the category.” As Nielsen notes, “until people begin to grasp the complexities of search and develop skills accordingly, businesses that take such extra steps to help users find what they need will improve customer success — and the bottom line.”

Million Short: A Search Engine for the Very Long Tail

Scott Gilbertson — Fri, 04 May 2012 18:01:47 +0000

Where is that needle? Photo: Perry McKenna/Flickr.

Imagine a search engine that threw out the web’s top one million sites and then searched what was left. Sounds insane, right? But that’s exactly what Million Short purports to do and the results are, well, interesting.

Million Short seems like a terrible idea. Why would you want to remove the top sites on the web from your search results? In most cases you wouldn’t, but what Million Short offers is a chance to discover sites that just don’t make it to the top of the results from more popular search engines like Google, Bing or even DuckDuckGo.

It could be that these missing sites are just small, or perhaps they don’t use cutthroat SEO tactics to compete for popular terms, or maybe they just cover topics so niche they’re unlikely to rise to the top of any but the most targeted of searches. It could also be that they’re content farms and other worthless pages. Whatever the case, skimming the top million sites off the web just might open your eyes to how narrow your filters (and Google’s) have made your results, and how that’s both good and bad.

As Million Short notes, popularity is not an inverse corollary to quality, but when the same popular sites show up over and over in your results you are inevitably missing out on something. And that’s what Million Short wants to show you.

It’s important to realize that Million Short is removing the top websites not just the top search results for individual queries. It’s also worth noting that Million Short doesn’t disclose where its search results are from, nor how it calculates the top sites. [Update: Sanjay Arora, founder of Exponential Labs, tells Webmonkey that Million Short is using "the Bing API... augmented with some of our own data" for search results. What constitutes a "top site" in Million Short is determined by Alexa and Million Short's own crawl data.]

Most of the time, narrowing search results down to trusted, well-known sites like Google, Bing and other search engines do is a good thing. To see why just plug a few programming queries in Million Short and you’ll quickly realize just how helpful Stack Overflow — well inside the web’s top 1 million sites — has become. At the same time you might discover some unknown blog that will never make the top results in Google and happens to have the answer to exactly your problem. Is that better than the same answer from Stack Overflow? That’s up to you.

Million Short does offer some customization options you can use to both cut out the top sites and keep the handful you don’t want to be without. Additionally you can change the limit from the top million to the top 100,000, 10,000, 1,000 or 100 sites. If you decide you love it there is a search engine plugin that will work in Firefox, Chrome and Internet Explorer.

Perhaps the better way to think of Million Short is not so much a search engine, but a discovery engine. Million Short’s strength is not going to be answering the specific kind of queries that Google is forever optimizing its index to handle, but to discover less well-known sites and explore the more remote corners of the web that might be lost in other search indexes.

Build a Custom Site Search Engine With ‘Tapir’

Scott Gilbertson — Fri, 13 May 2011 14:56:27 +0000

If you’ve switched from a dynamic publishing tool like WordPress to a simpler, static site — whether to take advantage of cheap Amazon S3 hosting, or because you want to publish from flat files, without a database — there’s a few things you may be missing.

Some content is necessarily dynamic. If your site is just flat html files with no database behind them, there’s no easy way to build comments, contact forms or built-in search indexes. Luckily the web has a few solutions. For comments there are JavaScript solutions like Disqus or IntenseDebate, and contact forms can be built with Wufoo, but search is a little more difficult.

You could use Google’s Custom Search Engine tools, but then you’ll need to display things on Google’s terms (including a logo). Yahoo has a similar offering, but its results are often sub-par. The lack of search options for static sites led developer Jeff Kreeftmeijer to create Tapir, a JSON search API that indexes content from your site’s RSS feed.

Designed with static publishing systems in mind (like the popular Ruby on Rails tool, Jekyll), Tapir handles search through RSS and JavaScript without the overhead of a database on your own server. Tapir offers a JSON-based API and relies on Tire behind the scenes (which is powered by Elasticsearch, which in turn is powered by Lucene).

To use Tapir all you need to do is write a simple JavaScript-based search form, query the Tapir index for your site and then parse out the results to display for your visitors.

Tapir will parse and store the RSS feed you supply roughly every 15 minutes. For older posts (i.e. posts already long gone from your RSS feed) you’ll need to use the API to send over the data — something of a pain, but at least it’s a one-time pain.

If you’d like to give Tapir a try, just head over to the site, sign up for a token and read through the basic API docs for details on how to implement your search engine. The Tapir website says that sample code and better reference materials are coming soon, ~~along with a JQuery plugin~~[Update: As Tapir creator, Kreeftmeijer, notes in the comments below, the JQuery plugin is now available].

See Also:

Google Uses HTML5, JavaScript to Visualize Popular Searches

Scott Gilbertson — Fri, 10 Dec 2010 23:46:06 +0000

Google has released its annual zeitgeist report, a look at how the world searched in the last year. The zeitgeist is Google’s record of popular search terms and draws on sources like Google Insights for Search and Google Trends. It’s also a reminder that, in addition to tracking you in the usual creepy ways, Google often reveals some interesting data.

The results are predictably disappointing — despite a year’s worth of events, Chatroulette and Apple’s iPad top the list of most popular searches — but the data visualization Google has created is impressive.

The visualizations combine HTML5 with some fancy JavaScript (which appears to rely on the Dojo framework) to offer maps, bar charts and timelines. The map is particularly cool, plotting out bar graphs of searches by country with an interactive timeline slider to narrow the results by month.

Other views include bar graphs of the top search terms by category. When you click on an individual bar, the graph morphs into a timeline.

There’s also a video with some overly-nostalgic music that walks you through the top terms of the year. Check it out:

See Also:

Firefox 4 Adds Bing to List of Search Engines

Scott Gilbertson — Thu, 07 Oct 2010 17:37:13 +0000

Mozilla has announced that Microsoft’s upstart Bing search engine will soon become a default part of Firefox’s search bar. When Firefox 4 arrives it will feature some slight changes to the list of included search engines, offering, in order: Google (default), Yahoo, Bing, Amazon, eBay and Wikipedia.

Bing is a new option, though savvy users have long been able to install a Bing search plugin on their own. Now, it will be much easier to access by clicking on the drop-down list in the browser’s built-in search box.

Microsoft’s search engine continues to make inroads against Google, and while Microsoft has had a search product for years, it’s taken a long time to make its way onto Firefox’s short list. Mozilla vice president of products Jay Sullivan says Bing’s inclusion now is based on its “significant rise in popularity over the past year.”

Google’s engine will still be the default option for Firefox users. Google remains a primary source of income for the Mozilla — the two companies share the revenue generated by Google searches typed from within Firefox’s search box.

The new search engine default list removes the Answers.com and the Creative Commons search engine choices. Answers.com is disappearing because, according to Mozilla, “we have heard from our users that Wikipedia is more useful as an included reference search engine.”

The Creative Commons search engine is being removed because the search tool itself has changed from something that searches just CC licensed materials to a more general search engine that duplicates what’s found in Google, Yahoo and others. Mozilla is careful to point that the foundation “will continue to actively support [the Creative Commons] organization and mission through grants and joint programs,” but not, apparently, its search engine.

Of course users are still free to install any of the thousands of search plugins for the sites they’d like — we’re fans of the Flickr CC search plugin and the Speckly torrent search plugin — but making the default plugins list means more traffic for those lucky sites.

In Bing’s case it also means an important new avenue to perhaps pull a few users away from Google.

See also:

Add a Google Search Box to Your Site

Webmonkey Staff — Tue, 16 Feb 2010 01:45:47 +0000

Unless you’re incredibly handy at writing complex algorithms, building a search engine for your website is pain. And in the end, yours probably isn’t going to be that great, even after all your hard work. So why bother? Especially when there’s already a reasonably popular search engine by the name of Google — maybe you’ve heard of it? — that’s perfectly willing to handle the job for you.

The Google Search API is not only really good at searching, since it accesses the Google index, but it’s also really easy to use.

The potential for search-based mashups is nearly limitless, too. But in order to learn how it works, we’ll confine ourselves to a much more common use case — a site-specific search engine for your blog.

Getting Started
1. Implementing The Basic Search Engine
Site Specific Search Engine
Where to go From Here

Getting Started

The first step is to get a Google Search API. Just login to your Google account and head over the application page. Tell Google the domain where you’ll be using the Search API, then copy and paste your key. We’ll need it in just a minute.

First, just to ensure there’s no confusion, the only search API from Google uses Ajax. There was an older SOAP-based API, but sadly, that’s no longer available. You might still run across a few SOAP-based implementations since Google hasn’t shut it down, but it doesn’t hand out new keys.

Tip: The other thing to keep in mind is that if you’re launching a new site, the site-specific results won’t exist, since Google probably hasn’t crawled the URL yet. If you don’t have one, set up a Google Webmaster account and tell Google about your site by creating a sitemap. That should speed up the indexing process, though you will likely still have to wait a few days before a Google search returns anything useful.

Implementing The Basic Search Engine

The first thing to do is open up your site template and add this line to the head tags:

Paste in that API key you generated earlier and you’re ready to go. For now we’re going to write all our code in the page tags. But if you end up with a long and complex script, it’s a better idea to break it out into its own file.

Next, add this to the :

What we’ve done here is create a function that fires up when the page loads and creates a new GSearchControl object, which is a text input box and a search button. There’s also a little “Powered by Google” badge that will appear. We then create a searcher. In this case we’re just using a normal GwebSearch, which mimics the Google homepage.

Other options include video search, image search, blog search and several other of Google’s specialized search engines. For more details check out Google’s search documentation.

Once we have the object initialized and the type of search set, we have to tell Google where to draw the object. In this case, we’ll use a

with an id of “search-box,” so add this code in the body of your HTML file wherever you want the box to appear:

Your users can now search Google without leaving your page. But that’s not exactly what we want. Read on to find out how we can limit the search to just your site.

Site Specific Search Engine

To restrict the results to just your domain, we need to create a site restriction. To do that we’re going to change this line:

searchControl.addSearcher(new GwebSearch());

To this:

var siteSearch = new GwebSearch();

siteSearch.setUserDefinedLabel("YourSite");

siteSearch.setUserDefinedClassSuffix("siteSearch");

siteSearch.setSiteRestriction("example.com");

searchControl.addSearcher(siteSearch);

Just fill in your site name and URL and you’re done. Give it a shot and you should see results limited to your domain — assuming Google has indexed it already.

Tip: You can string together as many of these site searches as you’d like and use setUserDefinedClassSuffix to add a different class to each domain. This makes it possible to do some fancy CSS work too, like color-coding your results by domain.

You can also create a search using a custom search engine if you have one defined. See the Search Docs for more details.

Where to go From Here

We’ve really just scratched the surface of what you can do with the Ajax Search API, so definitely read through the documentation and have a look at some of the examples. Mashup potentials abound — especially when using some the specialized search engines like local search or video.

Other options include the ability to control most of the look and feel via stylesheets, the ability to search Google Books to find quotes for your blog and more.

Tip: If this is just too much code (Aw, come on, really?) you can always use the handy Ajax Search Wizards to generate some cut and paste code that will perform basic searches.

Google Crawlers Now Understand ‘Canonical’ URLs

Scott Gilbertson — Wed, 16 Dec 2009 11:25:05 +0000

Migrating a web site from one domain to another is never easy. You’ll probably lose whatever Google ranking your old pages had, possibly break incoming links and generally disrupt the flux capacitor of the web.

Of course, there are occasionally good reasons to move your content and now there are some new ways to let Google know what you’re up to. The Google Webmaster blog recently announced that Google will support the cross-domain rel="canonical" link element. That means you can effectively migrate your site to a new domain even if you don’t have server access to do redirects.

In most cases, Google still suggests that, if possible, you use 301 permanent redirects to point both visitors and search engine bots to your new domain. However, if that’s not possible for some reason, (for example, if you’re migrating from a hosted blog service to your own domain) then you can add rel="canonical" element to your page headers and Google will index the new URL.

Note that in our example — moving from a hosted blogging service to a self-hosted domain — it’s OK if there are some differences between the new and old pages, but the basic content (the blog post) should be the same.

Previously, Google would look down on cases of duplicate content across domains. Given the number of content-stealing “splogs” out there, filtering duplicate content by domains is a good way for Google to stop search engine spam. The problem is there are legitimate reasons to have duplicate content, like migrating a site to a new domain, and now there’s a way to do it.

One important note, Google no longer recommends blocking access to duplicate content on your website, whether with a robots.txt file or other methods. Just use the rel="canonical" tag instead.

See Also:

Google Tests Redesigned Search Page

Scott Gilbertson — Wed, 25 Nov 2009 10:36:46 +0000

Google’s new look? The search giant is testing a revamped results page. Click the image for a larger view.

Google appears to be testing a possible redesign of its iconic search page. Whether or not the new prototype will ever become official remains unknown, but thanks to some clever JavaScript you can check out the new look today.

The Google watchers over at Google Blogoscoped have found a snippet of JavaScript you can paste into your browser’s URL field which will activate the new look. Because the JavaScript code sets a new cookie, you’ll most likely need to log out of your Google account before it works.

Once the cookie is set, refresh the Google homepage and you’ll see the changes. The search buttons have become blue and the overall look is a bit like that of Google Wave. More significant is the redesigned search results page (seen above) which features an always-on sidebar for narrowing search results by type, date and view.

The brighter, more Wave-like look of the prototype doesn’t bother us, but we’re not so sure about the sidebar, especially given that the same options are already available in the infinitely more compact menu that runs along the top of the page.

There is one new search option in the sidebar that you won’t find on the current Google page — the ability to see results from online forum sites.

The good news, should the new look utterly disgust you, is that so far Google hasn’t even mentioned the new look (and had not responded to our inquires when this story was published) let alone taken any steps toward making it official. Given Google’s track record of beta testing, we suspect the redesign will be thoroughly and publicly tested before it goes live, if in fact it ever does.

See Also:

Google Dashboard: One Service to Rule Them All

Scott Gilbertson — Thu, 05 Nov 2009 15:42:54 +0000

If you’ve ever wanted to see all the Google services you use — and how you’re using them — in one spot, then the new Google Dashboard is exactly what you’ve been looking for.

Google Dashboard is a one-stop shop for browsing through of almost all the Google services you’re using and, by extension, shows you everything Google knows about you. The nice thing about the new dashboard is that it gives you central way to manage and control that data — change privacy settings, control sharing and limit what data Google stores about you.

Each service listed in your dashboard contains an overview of your usage and links to change any data-sharing settings, edit any associated profiles and control who can see what. For example, the Google Reader entry in the dashboard shows a summary of your feeds, starred items and followers, and includes handy links to control your sharing settings.

There’s nothing in dashboard that can’t be found within the individual services themselves, but navigating through dashboard is considerably easier than trying to do the same on a service-by-service basis.

That said, Dashboard has a few quirks. For example my dashboard says I’m sharing a photo album on Orkut, but in fact it’s just the default album associated with my Orkut account, and it doesn’t actually have an photos in it. Ditto for my Picasa account.

Dashboard doesn’t currently offer any transparency about how your data is being used by Google for advertising or user-behavior data-collection purposes. It also offers little info about how (or how long) your data is being stored. It would also be nice if the Dashboard gave you a nice link to export all your data for each Google service. Eventually we’re hoping Google’s Data Liberation Front will fix that oversight and integrate some exporting tools directly into Dashboard.

Dashboard doesn’t currently support every Google service, though it does cover the most popular tools. The big omissions are Maps and Groups, though Dashboard does at least offer links to the services it doesn’t track.

To access the new Dashboard features, just click the My Account link in any Google service and then look for the new Dashboard link. Alternately you can head directly to the new Dashboard URL: https://www.google.com/dashboard.

To see Dashboard in action, check out the following video from Google:

See Also:

Webmonkey » search

Google Discontinues Site-Blocking Service

You Suck at Search

Million Short: A Search Engine for the Very Long Tail

Build a Custom Site Search Engine With ‘Tapir’

Google Uses HTML5, JavaScript to Visualize Popular Searches

Firefox 4 Adds Bing to List of Search Engines

Add a Google Search Box to Your Site

Contents

Getting Started

Implementing The Basic Search Engine

Site Specific Search Engine

Where to go From Here

Google Crawlers Now Understand ‘Canonical’ URLs

Google Tests Redesigned Search Page

Google Dashboard: One Service to Rule Them All