Archive for the ‘test’ tag
Here’s a simple but not easy question: how subject to cyclicality is your marketing?
Human beings are naturally cyclical in nature, because that’s how the planet around us operates. We do things differently when it’s summer weather than when there’s a foot of snow on the ground. That’s so obviously logical that it shouldn’t need to be pointed out. Strangely, many marketers forget this basic truth when they design their marketing programs and instead assume a static customer who does the same thing all the time.
Here are two quick tests to examine whether your business is experiencing any level of cyclicality. First, go to Google Insights for Search, switch to time range, choose the last four years, and type in the top search term for your business. Here’s an obvious example of cyclicality in the searches for iced coffee:
It should be absolutely no surprise that search volumes for iced coffee go up when the weather gets warmer. Go look at search traffic for your own business for the last four years and see if there’s any cyclicality in it.
Second test: go into your web analytics and download the monthly dataset for as long as you have data. Create charts that do exactly the same thing – show you year over year website traffic. Again, look for cyclicality. For bonus points, repeat with funnel metrics like conversions, closed sales, and revenue.
Is there a cyclicality to your search results from test #1 that you don’t see in your website traffic or business data? If so, you may be missing business opportunities that your audience is looking for that you’re not providing!
Want to read more like this from Christopher Penn? Subscribe now:
I recommend & use:
SEOMoz SEO software.
I covered this at Search Engine Land last night, but it is worth covering here as well.
When you do any A/B testing or Multivariate testing with your web pages or web site, it is important that you do not confuse GoogleBot. If Google thinks you are breaking their guidelines or they just think something is off, it can have serious repercussions on your search rankings.
Susan Moskwa wrote a detailed blog post on how you can do A/B testing or Multivariate testing and reduce the risk of losing rankings in Google. In short, she said do not cloak, use 302 redirects (not 301s), use the rel=”canonical” attribute and do not run the test for too long.
Here is her advice:
- No cloaking.
Cloakingâ”showing one set of content to humans, and a different set to Googlebotâ”is against our Webmaster Guidelines, whether youâre running a test or not. Make sure that youâre not deciding whether to serve the test, or which content variant to serve, based on user-agent. An example of this would be always serving the original content when you see the user-agent âGooglebot.â Remember that infringing our Guidelines can get your site demoted or removed from Google search resultsâ”probably not the desired outcome of your test.
- Use rel=âcanonicalâ.
If youâre running an A/B test with multiple URLs, you can use the rel=âcanonicalâ link attribute on all of your alternate URLs to indicate that the original URL is the preferred version. We recommend using rel=âcanonicalâ rather than a noindex meta tag because it more closely matches your intent in this situation. Letâs say you were testing variations of your homepage; you donât want search engines to not index your homepage, you just want them to understand that all the test URLs are close duplicates or variations on the original URL and should be grouped as such, with the original URL as the canonical. Using noindex rather than rel=âcanonicalâ in such a situation can sometimes have unexpected effects (e.g., if for some reason we choose one of the variant URLs as the canonical, the âoriginalâ URL might also get dropped from the index since it would get treated as a duplicate).
- Use 302s, not 301s.
- Only run the experiment as long as necessary.
The amount of time required for a reliable test will vary depending on factors like your conversion rates, and how much traffic your website gets; a good testing tool should tell you when youâve gathered enough data to draw a reliable conclusion. Once youâve concluded the test, you should update your site with the desired content variation(s) and remove all elements of the test as soon as possible, such as alternate URLs or testing scripts and markup. If we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines and take action accordingly. This is especially true if youâre serving one content variant to a large percentage of your users.
One thing she adds is that if you follow the recommendations above then it should have “little or no impact on your site in search results.” But she goes on to explain that there may be an impact do to content changes.
Forum discussion at WebmasterWorld.
Image credit to ShutterStock for comparing image
According to recent reports, Facebook is testing a new feature that will allow users to easily turn Facebook photos into printed postcards. The new feature, which is named “Mail a Postcard,” was initially built during a Facebook Hackathon and is only available to a few test users at this time. If a Facebook user is [...]
Felix Baumgartner and the Red Bull Stratos team will have to wait a little longer to leap from the edge of space.
After a successful second test jump from over 96,000 feet last month, Red Bull announced today that Baumgartner’s final jump from 120,000 feet has been pushed back to October. Originally scheduled for this summer, the capsule that carried the famed BASE jumper during his 90-minute ascent suffered damage after landing on some rough terrain.
During his last jump, the Austrian BASE jumper reached speeds of 536.8MPH during his 3 minute and 48 second free fall becoming the second highest jump in history. Come October if all goes according to plan, Baumgartner will attempt to break Joe Kittinger’s world record of 102,000 feet, which the retired Air Force officer and Stratos advisor set in 1960.
According to Red Bull, the “outer shell, framework and other key components” were found to be compromised after ten days of testing but that the inner pressure sphere and other support systems remained intact. Once the capsule is reassembled it will be shipped out to Brooks-City Base in San Antonio, where it will be tested in an altitude chamber that replicates the stratospheric environment in late September.
Baumgartner and co. will attempt to create history in the first two weeks of October when both the weather and wind will be favorable.
Google announced they are having an opt in field trial of including Gmail emails, contacts and flights in the right hand side bar of the Google search results…
In response to an Apple Support Communities forum thread, a report on Tuesday offers anecdotal evidence that the new OS X 10.8 Mountain Lion may be causing battery life issues for some MacBook users.
Hey, here are some personalized search results that I might actually like.
Google has been expanding its Universal Search feature for a while, adding images, news stories, and more to its search results. Today the company announced that it’s starting a field trial that will experiment with adding your personal Gmail results as well.
After all, a lot of useful personal information is now piling up in your inbox, so Google plans to make that information available in your web searches (assuming you’re signed in to your account). For example, if you need to access your flight information while you’re checking in, you don’t need to open up Gmail — you can just search for “my flights” from Google, and you’ll get a list of relevant information from your email appearing on the right side of the screen.
Google’s Sagar Kamdar also demonstrated using this feature to check on his Amazon orders to see if a package has shipped yet, and he talked about other services that might be integrated in the future. Imagine not just searching for your OpenTable reservations, but also connecting that information with other data that Google has access to, such as getting reminders of when you need to leave if you want to arrive at a restaurant in time for your reservation.
In the current test, Gmail results are collapsible, and, along with Google’s other personalized results, they can be turned on and off. Google also demonstrated a mobile interface, but cautioned that there will be challenges to making this work on phones while maintaining SSL security.
This comes at a time when Google has been adding a lot of personalized and social features to search, usually tied to Google+, and often meeting with a mixed response from users (or at least from me). While it’s hard to judge the results without playing with the feature for myself, but this sounds like an example of how using other Google services to personalize search can be genuinely useful.
Senior Vice President Amit Singhal said the field trial will be open to 1 million users initially. He said Google would consider integrating this with other email services, too.
The Gmail field test was announced at a press event in San Francisco, where Google also previewed some improvements to its Knowledge Graph feature and an update to its search app for the iPad.
You can sign up for the field trial here.
This is a personal request from your user, a rallying cry from a compatriot. I personally love WordPress. I make my living from it. The average user, though, couldn’t care less about it. They just want to run their business, tell their family history, organize their church, share their photos or live their life online with a minimum of impedance. In its evolution from simple blogging tool to CMS, framework and software ecosystem, WordPress is losing its way. It needs us to help bring it back and cultivate simple genius.
My agency married WordPress in 2007. We’d been dating for a number of years but were still seeing others: some serious flirtation with Joomla, a blind date with Drupal, a summer romance with CMSMS, even a steady five-year stint with a custom CMS that we lovingly named Rocinante (after Don Quixote’s loyal steed). We tied the knot with WordPress for one single reason: about six to nine months after most of our projects, we would get the fateful call. “The only person who really understands how to use the website you built just left the company, and we need someone to train us!” It was almost inevitable, except on WordPress. No one ever called for help after a WordPress project except to share their excitement and book the next project. They just figured it out. It was easy and obvious and beautiful. Our clients loved it, and that was something you could grow a business on.
Then, WordPress started to grow up. New features like the menu manager, theme editor and sidebar widgets made WordPress more robust but more complicated. The ecosystem of plugins exploded. WordPress plugins are harder to use than they should be. Ask your users. We did. It was quite illuminating and a hint embarrassing. We decided to act on Tom Ewe’s call to arms and lead by example:
“I find it astonishing that WordPress developers haven’t worked harder to create usability guidelines for plugin development. Even experienced WordPress users are often left guessing as to where they should go to work with a new plugin.
One of the key drivers of WordPress’ success has been plugins, and yet they are not actually that easy to use. They appear as being stapled onto WordPress, as opposed to integrating seamlessly. Surely there should be some common usability rules when it comes to plugin development?”
— Tom Ewe
We Lack Conventions, And This Is Why It’s A Problem
Three weeks ago, we brought on Joyce to our customer team at Modern Tribe. She’s smart, she has a real power-user’s/light themer’s grasp of WordPress, and she had never used our free WordPress.org-hosted plugin, The Events Calendar, nor any other of our add-ons. She came back after looking them over and said, “This is far harder to set up than it should be.” I asked her whether she had read the new user primer or the set-up instructions. “No, I didn’t. I bet most of your users don’t either.” I had to admit that Joyce was probably right. Rather than try to list all of the things that she thought might or might not work, she pointed me to Steve Krug’s SxSW talk “Rocket Surgery Made Easy.” I couldn’t turn it off. I’ll boil it down to a few paragraphs for you, but if you develop a plugin or theme or have a product business, this is a must hear.
Krug argues that hiring usability experts is unnecessary (heck, let’s be honest: most of us don’t do it anyway). The real value of a usability test is in getting together (ideally with sushi) and observing the experience, not hearing an expert’s interpretation. Within 15 minutes of watching the first user try to use our plugin, a handful of long-running arguments were resolved and some incredibly simple hurdles were exposed. I’ll walk you through the process that we followed for a remote usability test of The Events Calendar.
Our Remote Usability Test: Step-By-Step
- Total time invested: 6 hours
- Set-up: 1 hour
- Testing: 3.25 hours
- Notes: 0:45 minutes
- Team review: 1 hour
- Find three participants. We had enough users and visitors that a blog post generated about 15 willing offers. We gave away a free copy of The Events Calendar Pro in exchange for participation. Make sure that the criteria for participation are explicit. Krug insists that you really don’t need more than three users, and that turned out to be spot on. By the third user, we were accurately guessing where they would fail. Schedule the test to last about 30 minutes to an hour, depending on the tasks, and give yourself time in between to clean up your notes and deal with other details.
- Think of some process or features you want to explore. We were curious to see how first-time users experience our core Events plugin. With that in mind, we made a series of nine steps that we knew were pretty common for setting up the calendar. Make sure to write them out, and give goal-based instructions, not actual steps. Think, “Create a new event,” rather than “Click the new events menu to make an event.”
- Set up a domain with WordPress and your plugin or theme on it. If you are testing a plugin, decide whether the problem or feature set that you defined in step 2 is best served by a fairly vanilla build (for example, 2011 theme + minimal plugins + no content) or by a more real-world build (perhaps use your demo content if you have one or a user’s website backup). Configure the whole website precisely for the first step. Run through it once entirely to make sure you haven’t forgotten anything obvious.
- Back up the database of the website so that you can restore between tests.
- Grab a copy of Join.me or your favorite screen-sharing or VoIP tool (such as GoToMeeting or Adobe Connect). We found that Skype just wasn’t stable enough to carry us through the screen-sharing portion of our test run. Join.me functioned amazingly well, except for an issue with voice echo caused by laptop sound cards during one test. The fact that it was free was appealing. Make sure that both screen-sharing and voice are available in whatever set-up you choose and can be recorded together. We used ScreenFlow to record the test so that it could be reviewed later.
- Do a quick test run with someone on your team (or your mom), and make sure that the kinks are worked out.
- Get the whole team ready and present. Do whatever you can to get people to participate. Everyone on our team who participated was blown away by the experience. Buy them fancy snacks or digital beer. Fire up a chat session if your team is remote (one that the test participant is not privy to) so that your team can chat freely. If you are co-located, make sure the team is not in the room where the test is taking place. Twelve people hovering over someone’s shoulder will unnerve even the most confident person.
- The introduction and set-up are key. Krug has a great script that we just followed. The first key: explain to the participant that the plugin is being tested, not them. There is no wrong or stupid choice. If something is hard or confusing, it’s our fault and we apologize. Secondly, encourage the participant to speak out loud and share their thoughts; i.e. provide a guided monologue. Give them a copy of the steps (paste them into the chat session or email them beforehand), and read them through together once.
- Read a step. Watch. Shut up (bite tongue). The goal is to watch them as if you weren’t there, so don’t help them. This can get crazy awkward, but observing the various choices they make in trying to accomplish a goal becomes very informative. Consistently ask questions to get them to speak out loud, such as “What are you thinking?” and “What did you expect?”
Observing user 2 figure out where to add events to her menu. (Large version)
- Have the moderator and the people observing take notes on what they see, and discuss together.
- Once all of the steps were completed, we asked a bunch of probing questions. We were surprised by how much two users employed the admin bar, so we asked more about that. We were curious why no one clicked the tutorials, despite having the answer in the title. And on and on.
Usability has to do with more than what’s in a plugin’s admin settings. We probed why none of the users took advantage of the tutorials. It turned out that a blog loop has no useful organization, so we made a quick page to group the posts by topic. (Large version)
- Time to pay the participant in money, karma or free goods and get ready for the next test. Reset the website’s database.
- Take some time to condense your notes. Ask everyone who observed to pick the three most important things that can quickly be fixed based on the test. The goal is not to do a redesign; we are looking for quick course corrections. Then we test again in a new cycle.
Notes were broken down into observations, user recommendation and bugs. (Large version)
Findings From Our Tests
A number of our major debates were instantly answered. For example, we had had a prolonged disagreement about the placement of the menu item for the plugin’s settings. The majority of the development team felt that it belonged in WordPress’ main “Settings” tab because that is a de facto standard. A minority of developers and all of the community team thought that putting it in the submenu for the Events custom post type would be more intuitive.
Both sides had great arguments. For the test, we put it in the WordPress settings, and then we watched three users in a row fail to find it there in a reasonable timeframe. One found it from the top admin toolbar (we put it there, too), one eventually looked in WordPress’ “Settings,” and one gave up despite looking right at it three times. Standards are great, but we all had to admit that functionality has to supersede a poor standard. We explored putting it in both places, but ultimately we decided to move it to the Events menu for now due to technical limitations.
We also saw how hard a time users had finding the events calendar on the front end of the website, despite it being in five locations. By seeing where people looked for it, we came up with a game plan that took five minutes to implement, and we hope it will make it a whole lot more intuitive.
The usability test was so valuable that Paul, one of the developers, asked if we could do it every month. Usability testing has, without a doubt, provided the best feedback we have ever gotten on our product, it cost very little, and it has now been added to the monthly production schedule. We will be testing these updates next week to see if they truly did improve the experience.
I’m continually amazed by a community’s ability to reach the same conclusion at the same time. Last week, Dave Martin posted for the first time to the core UX team’s blog:
I’m just getting my feet wet, and quite honestly haven’t a clue where to get started, so I thought I’d set up a quick user test (I’m a big fan of user testing). I set up a temporary WP install, and ran a user from usertesting.com through a couple scenarios.
Check out the video. It almost hurts to watch her struggle. I can’t tell you how grateful I am to see the core team paying attention as well and engaging quickly. It is a great start.
Call For WordPress Human Interface Guidelines
The average website has over five plugins installed (according to PressTrends) and often a theme options panel. For a great experience to continue throughout the website as people actually experience it, we need to establish strong standards for the rest of the community to follow.
I am calling all WordPress plugin developers and themers. You don’t need to guess what your users might want or how they will experience your product. Just watch them. We know it: if we focus on usability, stability and then value, we can make products that users will line up for.
To the core WordPress team and the community at large: Let’s get together and create WordPress human interface guidelines for those who contribute by providing plugins and themes for the world to use. Apple gave us a rock and upon it built a foundation that few can deny. Google finally got around to it with Ice Cream Sandwich, and I expect to see drastic improvement in the wild west that is the Android application landscape. Help us help WordPress.
In the words of Matt Mullenweg when he saw Dave’s first post:
Thank you very much for this, I think more frequent and more transparent testing will allow us to make much better informed product and UX decisions. If we do this right we should see the videos get better and better (shorter and less confusion) from release to release.
Code is poetry. So should be your user’s experience.
© Shane Pearlman for Smashing Magazine, 2012.
Yesterday, Bryson Meunier spotted a new test in Google’s mobile search results where they label smartphone-optimized pages with a little smartphone icon in the search results. Here posted some low quality images but Yoshihiko…
Yesterday, Bryson Meunier spotted a new test in Google’s mobile search results where they label smartphone-optimized pages with a little smartphone icon in the search results. Here posted some low quality images but Yoshihiko Yoshida has some better quality images which I am sharing here:
Google confirmed this test telling me:
Weâre experimenting with ways to optimize the mobile search experience, including helping users identify smartphone-optimized sites. We donât have any more details to share at the moment, but thank you for checking in.
Since Google offered up their official recommendation on mobile SEO, this is a good indicator (if you see the test) to see if your site is indeed implemented properly for Google to recognize that your site is smartphone optimized or not. There is no such feature in Google Webmaster Tools as far as I know. There are mobile sitemaps, but that is used for feature phones.
I am also pretty sure Google had a similar implementation, of labeling phone friendly pages in the search results, specifically for feature phones. Moving it to smartphones is neat, although I am not sure if Google will add this clutter to the page or not in the future?