Build Content Topics, Fast with Auto Suggest

Today I’m going to share with you a really useful Python script to get you started with scraping some of the Auto Suggest API’s.

In a lot of ways, this is limited only by your imagination. Personally, I like to use this for Q&A style comments – Great for building out FAQ’s for e-commerce sites, or general content building via ‘topical keyword research’. Let’s cut the waffle and jump straight in.

Say you have an online Wine Shop. With the following script we can quickly gain access to common searches from potential customers, the key here is that we’re specifying ‘question’ leading keywords; in this example, ‘what’, ‘who’, ‘how’, ‘does’, ‘why’, ‘can’, ‘where’, ‘when’ – We’re also including the wildcard operator to help plug any holes in-between our query and our ‘question’ keyword, specified within our script as ‘variations’. Read on and it will make sense…

# -*- coding: utf-8 -*-
import requests
import json

# here you include your main terms, this can be as many as you like
terms = ['wine']

# these are just 'variations' which are appended to the start of your 'terms'
variations = ['what * ', 'who * ', 'how * ', 'does * ', 'why * ', 'can * ', 'where * ', 'when * ']

for term in terms:
	for variant in variations:
        # request api
		r=requests.get('http://suggestqueries.google.com/complete/search?client=firefox&q={}{}'.format(variant, term))
        # grab returned text loading as json
		parsed_json = json.loads(r.text)
        # print each parsed item
		print parsed_json[0]
		for i in parsed_json[1]:
			print '- ', i.encode('utf-8', 'ignore')

When running the script we get the following (we’ll look at these more closely, shortly).

If  you want this going straight to a file, simply change the run command to the following:

python suggest.py > outfile.txt

From here, depending on what we are trying to do, we can use the information to our advantage, for example, we might want to group these into areas to create some epic long-form content. For example; with a little cleaning, we can create the following groups:

Food suggestions:
– what wine goes with turkey
– what wine goes with salmon
– what wine goes with pork
– what wine with ham
– what wine with lamb
– what wine with chicken
– what wine with duck
– why wine and cheese

Wine types:
– what wines are sweet
– what wines are dry

Section for the health conscious:
– what wine is good for you
– does wine make you fat
– does wine have carbs
– does wine have sugar
– why wine is good for you
– how wine is good for health
– does wine have gluten
– does wine have yeast
– can wine cause heartburn
– can wine make you fat
– why wine gives you a headache
– why wine is bad for you
– how wine is bad for you

Storage and testing:
– does wine freeze
– does wine expire
– does wine go off
– does wine go bad
– can wine be frozen
– can wine spoil
– can wine go bad
– can wine freeze
– when wine is opened how long will it last
– can wine expire
– when wine goes bad
– when wine is corked
– how wine is corked

DIY:
– how wine is made
– how wine works
– homemade wine

Passionate/Opinion pieces:
– why wine tasting is bullshit
– why wine is better than beer

Effects on the body:
– why wine makes me sleepy
– why wine makes you sleepy

History:
– wine who invented
– when wine was invented
– where wine was invented
– how wine was invented

Drinking tips:
– when wine has legs
– when wine tastes best
– when wine tastes like vinegar

Product information:
– who makes wine
– who ships wine
– where wine is made
– where wine grapes grow
– why wine bottles have a dimple in the bottom
– can wine get you drunk
– does wine get you drunk

With very little work we’ve pulled together sections for a long form content piece or multiple individual pieces. We’ve also got a good indicator about what people care about, for example, the section for the health conscious is fairly big in comparison to some of the others.

There are a few things to note here, firstly, some of the above groups could arguably be split into further groups- If I had a wine client I’d have put more effort in. The other thing to note here is to look past the way people are searching; for example, “wine who invented”, it’s typical searching habit but don’t be dissuaded; just group it into it’s category and take it as another indicator that you can create something someone is searching for (spelling is another big one, just ignore it).

So that’s it, I’ve hopefully given you something to think about. As a sweet bonus, here’s the API’s for Bing and Amazon…If you want me to do tutorials on these or show you some other tips and tricks, then let me know.