Or: The Robot Ate My Web Page…

 

Editor’s Note: In the second post in our series on optimizing your web pages for ad relevancy, Yahoo! Publisher Network Senior Product Manager Cody Simms and Senior Product Manager Amit Paunikar talk about how important it is to feed our “bots” a steady diet of tasty, easily digestible content. Cody and Amit also show how you can cook up more palatable pages to help keep our gormandizing little bots coming back for more—and help keep your ads relevant.

 

As we mentioned last week, content is king… but only if our content analyzers—the “bots” that read your text—can appropriately analyze your content. Here are some tips to help make sure our bots gobble up all of the good stuff.

 

Anatomy of a web page
Bots digest everything they eat, but only if they can eat it. They love text. In fact, they can’t get enough of it. If you want to help make your site compelling to a bot, focus on making it as text-based as possible.

 

Carving up a web page is not entirely unlike carving up a side of beef. Here’s a quick diagram showing the choice cuts that bots like to eat, and the ones they don’t:

 

 

In particular, bots struggle with:

• Images (Of course, your site does need images; just make sure that you use “ALT text” so that bots can nibble on them.)
• Multimedia
• Flash
• Frames
• Pull-down lists
• Fly-out menus
• Anything behind a log-in

 

If you want to avoid “robotic indigestion,” be sure to explicitly include important topic-oriented items in the text fields in your site, rather than hinting at them in images or other elements.

 

While you’re at it, check your robots.txt file to make certain that you aren’t blocking our bots. Check out robotstxt.org for more information.

 

There’s no place like home
Home pages, sometimes called index pages, are the face of your web site. First impressions matter to bots just as much as they do on first dates. But home pages can be sometimes tricky for contextual ad networks.

 

Most home pages have static content, and that’s bad. If the content is static, the ads related to the content will likely also be static. Static ads can lead to user fatigue—users get bored looking at the same content all the time, just like bots. This may result in low click-through rates, which can result in reduced revenue.

 

You can help remedy this situation by adding some dynamic content on the home page—content that changes frequently while remaining on-topic for your audience. New content perks up your audience, as well as our content analyzers.

 

Some home pages, especially on blogs and forums, have the opposite problem. They are too dynamic and can become topically inconsistent. This can result in ads that are all over the map. Using the Ad Targeting feature can help alleviate this issue, though it’s important to stay on-topic and on-target with your audience.

 

Site structure
Just like content, you can use your site’s structure to help your users and our bots understand what your site is about. In fact, site structure can be just as important as content for semantic analysis. Here are some tips to help you out:

 

Integrate keywords into your URL structure
One way to do this is to use “permalinks” rather than “query strings,” for example:

 

Permalink (good):
http://hotjobs.yahoo.com/educationjobs

 

Query string (bad):
http://hotjobs.yahoo.com/jobseeker/jobsearch/search_results.html?job_interest=EDUEDUCA&country1=USA&metro_area=1&basicsearch=0&advancedsearch

 

Use strong keywords as “anchor text”
Link text should be descriptive of the content that your user will find under the link, especially when linking to your own sites, both in navigation and in editorial content. For example:

“Education jobs,” not “Click here”

 

Use concise descriptors
It’s important to put a short description under a link, for context, where possible:

 

Education jobs
Search for jobs in the education field, including teacher jobs and administration jobs.

 

Next time: Writing good, bot-loving content, meta-tags, and keeping the noise down.

 

–Cody Simms, Senior Product Manager and Amit Paunikar, Senior Product Manager