Why is it important to be able to effectively test product ideas? What are the downsides of not testing or poor testing?
Effective product testing can help you optimize your business model – the most successful startup experience I’ve had began as a B2C company and pivoted to B2B. This pivot was a result of realizing that our product solved a problem for banks rather than consumers. Without testing, realizations and successes like this aren’t possible.
Avoid over-investing in zombies – many startups persevere with their ideas without testing them thoroughly. This leads to wasting resources on ideas that aren’t feasible or desired. If you jump straight to building a solution and trying to scale it without thorough testing, you may find there isn’t a need for the scale you’ve created. Your idea limps along with just enough customers to keep it alive, but never enough to make it lucrative.
Running Product Idea Tests
What are the steps to generating and testing a product idea?
| Stage 1: Idea Generation | |||
| Steps | Ideas are typically generated in two steps: • Reimagining an existing product or service – this involves looking at the value your product or service delivers in a new light. It’s a needs-based approach where you consider the tasks your customer segment is trying to accomplish, the challenges they face, and the gains they’re seeking. • Discovery – this step involves prioritizing the needs you’ve identified and gathering evidence to support the existence of these needs. This process will help you generate ideas that address these needs. | ||
| Tips | Gather evidence from customers – before investing heavily in an idea, make sure you have enough evidence to support it. This might involve talking to potential customers, conducting market research, or running small-scale experiments. If customers request a feature, understand the need behind it – when a customer requests a feature, it’s important to understand the job behind that feature. What is the customer trying to achieve? Getting at the need behind the request can prevent you from getting caught in a loop of building features that customers don’t use or pay for. Understand that people’s behavior may differ from their words – people might say they love or want a feature, but their behavior might tell a different story. It’s important to observe how people actually use your product or service, not just what they say about it. This can help you to understand whether your idea truly meets their needs. We call this the “say-do gap” and closing it is your key to success. | ||
| Stage 2: Testing your Idea | |||
| Steps | Testing your product idea involves a three-step process: • Extracting Assumptions – identify the assumptions in your idea. Prioritize these assumptions based on their importance and the amount of evidence you have to support them. • Testing Assumptions – test the riskiest assumptions first; the most important assumptions with the least amount of evidence. This process is iterative, and you’ll need to revisit and reevaluate your risks as you progress. • Iterative Testing – risks change over time. Initially, the risk might be that customers don’t need what you think they need. Later, it might be that they need it, but won’t pay enough for it. Finally, it might be that they need it, will pay for it, but you don’t know how to execute it well and scale it. | ||
| Tips | Avoid confirmation bias – it’s easy to focus on the positive feedback and ignore the negative when testing an idea. However, this can lead to confirmation bias, where you only pay attention to information that confirms your preconceptions. To avoid this, make sure to consider all feedback, both positive and negative. Testing usually takes about six to twelve weeks – if you run at least one experiment a week for twelve weeks, you’ll have enough directional evidence to decide whether to pivot, persevere, or kill the idea. This timeframe isn’t set in stone, but it’s a good guideline. Implementation could take months but in software, you can make significant progress by balancing discovery and delivery in six to twelve weeks. | ||
Avoid rushing into building – it’s easy to get excited about an idea and jump straight into building it. However, this can lead to wasted resources if the idea isn’t thoroughly tested first. It’s important to take the time to validate your ideas before investing heavily in them.
What is assumption mapping? How do you apply it to create effect tests for product ideas?
It is a strategic tool to identify and prioritize the riskiest assumptions in a business idea – it is a facilitated exercise that involves representatives from design, product, and engineering teams. The process is divided into three main categories: desirability, viability, and feasibility.
| Risk Category | What it is | Examples of tests to run | ||
| Desirability | Identifying the risks around the value proposition – the customer, their jobs, pains, and gains. | • Customer interviews • Surveys • Preference experiments • Comprehension tests • Open-ended discovery • Search trend analysis | ||
| Viability | Understanding the financial viability of the idea – determining whether people will pay enough for the product and whether costs can be kept low enough for the business to be profitable. | • Willingness to Pay (WTP) surveys • Price testing • Getting commitments in writing, • Pre-sales • Mock sales | ||
| Feasibility | Assessing whether the idea can be executed – includes evaluating technology risks, regulatory risks, and policy risks. | • Prototyping • Creating Minimum Viable Products (MVPs) • 3D printing | ||
What should you prioritize which risks you test first?
Once risks have been identified, prioritize testing the riskiest assumptions – based on their importance to business success and the amount of evidence supporting them. The riskiest assumptions, also known as Leap of Faith assumptions, are those that are most important and have the least amount of evidence. These assumptions are then tested through a series of experiments over a six to twelve-week period.
Typically, testing for most ideas efficiently follows three stages:
- Stage one: De-risking desirability – with early-stage ideas, the biggest risk is usually desirability, and the focus is on determining if there is a genuine problem to be solved. The initial testing phase is primarily about desirability, with some focus on viability.
- Stage two: As the idea gains traction, shift focus to viability testing – once there is evidence that people want the product or service, the next step is to determine if they will pay enough for it and if the costs can be kept low. This involves both internal and external testing to understand willingness to pay and the financial feasibility of the idea.
- Stage three: Feasibility testing – this involves figuring out how to build and deliver the product or service. It’s a common mistake to jump to feasibility testing first, skipping desirability and viability. This often leads to the realization that the market is not big enough, willingness to pay is not sufficient, or maintenance is too expensive. The aim of the progression is to defer feasibility testing as much as possible (as it’s often very expensive).
How do you balance vision with customer feedback?
Having a vision is crucial, but it’s equally important to test that vision against reality- this involves testing the vision in a small way initially and then gradually over time. The goal is to determine whether the vision is realistic, whether it can be built, and whether it’s worth building.
There is a “say-do gap” between customer feedback and behavior – companies have to remain conscious of the difference between what customers say they want and what they actually want. While it’s important to listen to customer feedback, it’s equally important to use your expertise to build something that solves their needs. This could involve creating a solution that customers haven’t even contemplated.
Close the “say-do gap” by testing if customers actually take action – this could involve asking customers for a Letter of Intent, to sign up for a service, to refer others to the service, or even to agree to another conversation on the idea. Light evidence, such as survey responses or customer interviews, can be useful but should be treated with caution. Avoid making significant business decisions based on this type of feedback alone. Stronger evidence like presales or LOIs can provide real support for a decision.
Verify feedback with tests on prototypes or beta sign-ups – before making a big investment based on feedback, consider creating a clickable prototype, a beta version of your product, or an explainer video. This allows you to gauge the respondents’ reactions and verify whether they would behave as they indicated in their feedback.
Give additional weight to feedback from strong relationships – if you have a strong relationship with a current customer and they suggest something tangential to your existing offering, consider this as stronger evidence than survey responses. You understand the context of their feedback and can trust their intentions more.
How do you know when an idea is ready to bring to market?
The process of bringing an idea to market involves continuous testing and investment – this could span weeks or even months and is crucial to understanding the viability of your idea. The decision to continue investing in an idea is often tied to the results of this testing. If the evidence gathered suggests that the idea has potential, then it’s worth continuing to invest in its development.
Keep in regular communication with company leadership – about the progress and results of your testing. This ensures alignment on whether to continue with the idea, pivot in a different direction, or stop altogether. It’s important that the team working on the idea and the company leadership are on the same page about the riskiest assumptions, what has been learned, and the recommendations moving forward.
How do you determine when an idea has failed?
Stop if your tests indicate an insurmountable roadblock to success with potential customers – this could mean that no one is willing to pay for your product or service, or that it doesn’t solve a significant problem for people. In such cases, it’s better to redirect your resources elsewhere.
How do you make funding decisions on your product ideas?
Begin with metered funding – particularly for early-stage ideas, it’s important to allocate funding in tranches. Instead of over-funding a concept at the idea stage, allocate a small amount to explore its potential. The amount can vary depending on the industry, but the goal is to conduct enough discovery to determine if there’s something worth pursuing with desirability experiments. If it shows promise, provide additional funding. This could be used for building MVPs (Minimum Viable Products) and conducting activities that involve a value exchange with a real customer.
Consider your product idea funding like an internal VC – this approach is similar to how venture capitalists operate. It involves running experiments, gathering learnings over a series of weeks, and presenting these findings to the decision-makers in the company. The decision to continue, pivot, or kill the project is made collectively, with the funding tied to the need to de-risk the project further.
Tie each funding tranche to a de-risking goal – always tie the funding requests to the tests you want to run. This ensures that the money is being used to de-risk the project further and gives an idea of how the funds will be utilized.
B2B Experiments
In what ways is it harder or easier to test B2B ideas vs. B2C ideas?
| Harder for B2B | Easier for B2B | ||
| You might have a smaller pool of test subjects – in B2B markets with a small number of potential customers, testers have to be careful. You don’t want to risk alienating your customers with failed tests. Tests have to protect your brand – in early stage B2C, companies can afford to be a bit more flexible with their brand image. B2B companies can’t make promises they may not be able to deliver, as it can tarnish your brand and damage relationships. You may have different customer profiles – you may have user customers, influencers, recommenders, economic buyers, and decision-makers—each with a different value proposition. You have to identify these, and test value propositions for each so you don’t end up with users who love your product, but decision-makers who are not willing to adopt. You have to overcome the culture of pitching – many B2B businesses hesitate to contact customers unless they’re pitching or selling them. This mindset is gradually changing as successful companies realize the importance of regular customer conversations. Build a strong relationship with customers where you can discuss early-stage ideas, show them mockups, and co-create. | You can co-create with your customers – unlike B2C markets where you can afford to experiment with different value propositions, B2B markets require a more collaborative approach. Engage your customers in discussions about their problems and potential solutions. This co-creation process can lead to more effective and customer-centric solutions—and you can talk about other problems you want to solve. Customers may come with stated and known preferences – B2B customers often have a list of issues they need to address. You can help them prioritize these issues and understand their preferences. This will allow you to focus on solving the most critical problems first—and avoid building solutions for minor nuisances. |
What are some of the best tests for B2B ideas?
| Sample B2B Testing Sequence |
| Experiment 1: Customer Interviews – to understand the needs, struggles, and gaps in the existing market. |
| Experiment 2: Discussion Forums – participating in or observing discussion forums about existing incumbents in the market. |
| Experiment 3: Boomerang Testing – also known as competitive user testing, involves having people go through the competitors’ products to understand their pros and cons. |
| Experiment 4: Clickable Prototype – this is a simplified version of your product that users can interact with to provide feedback. |
| Experiment 5: Presale – this involves selling a simplified version of your product or service to gauge interest and generate early revenue. |
| Experiment 6: Single Feature MVP – this is a version of your product that does one thing really well and you can build out from there based on customer feedback and needs. |
The key is to start with building lighter evidence and gradually creating concrete proof – you want to minimize risk of your investment and eventually get something in front of people that they can interact with and provide feedback on. This iterative process allows you to continuously improve your product or service based on real-world feedback and needs.
Conduct discovery with customer interviews and preference testing – one effective way to test in B2B markets is through customer interviews and preference testing. This approach allows you to gain deeper insights into your customers’ needs and preferences. It also helps you understand what’s most important to them.
Culture of Testing
How do you create a culture of experimentation, especially in an established B2B business?
Patience is key – cultivating a culture of testing and experimentation within your company takes time. This is particularly true in the B2B world where the norm is to avoid customer confusion until a pitch-ready product is available. Changing this mindset is a gradual process.
Recognize the sources of your ideas and prioritize those that are customer-centric – the origin of your ideas is crucial. Are they born out of ‘wouldn’t it be cool if’ moments or are they derived from actual customer needs and feedback? The latter is more likely to yield successful results.
Structure your organization and offer funding to nurture ideas – it can be beneficial to dedicate one or two people to early-stage ideas for initial vetting, discovery, and testing. This allows for focused attention and the development of a more refined concept. While you don’t want to overfund an idea in its early stages, it’s important to allocate enough resources for running experiments.
Build a library of testing methods and resources – if your team only knows how to conduct customer interviews, and surveys, and create landing pages, your learning is limited to these areas. Expanding your testing library to include things like clickable prototypes, explainer videos, and MVPs can greatly enhance your learning and development process.
Create an internal process for testing – this process should allow for a steady flow of ideas, quick risk assessment, and the ability to test these ideas. It’s also important to have dedicated personnel for this process.
How do you think about how much capital/what share of your resources to dedicate to testing product ideas?
Guidelines for divvying up company resources is:
- 60%-70% on core business – the majority of your resources should be dedicated to what you do best. This includes maintaining and growing your core business, ensuring that operations run smoothly, and keeping customers satisfied.
- ~20% on adolescent ideas – approximately 20% of your resources should be allocated to ideas that show potential but need further development. These “adolescent” ideas may not immediately generate significant revenue, but with nurturing and attention, they could become major contributors to your business.
- ~10% to experiment with new ideas – this is for pursuing completely new ideas, akin to deep-sea diving; you may have to dive several times before you find a gem. While most of these ideas may not work out, you might discover something that could revolutionize your business.
Give your team time – creating new revenue streams is not a trivial task. It requires a significant amount of time and dedication. Therefore, it’s important to give your team the time they need to fully explore and develop new ideas.
Should you test product ideas within existing operating teams, or should you create a separate “skunk works”?
Typically, product management owns testing and generation of ideas – they look at customer support data and sales data to understand customer behavior and market trends. This information can help identify potential opportunities that align with the company’s strategy.
If you can, dedicate a team to pursue new opportunities – it’s beneficial to have a dedicated team working on early-stage ideas. These are usually the creative individuals who are comfortable navigating uncertainty and have a knack for pulling projects out of the fire.
Allow team members to own their ideas – give team members the option to stay with their idea and own it, or to move on and work on a different idea. This gives them the permission to fully invest in an idea they believe in, or to explore other opportunities if they choose.
How do you facilitate seamless handoffs from your team handling experiments to your operating team?
Involve other team members in the testing – start involving other team members and socialize the ideas. This helps to ensure everyone has the context for the testing and understands the why behind the work. When a small team vets out ideas and then throws them over the wall to another team, a lot of context can be lost.
Document business model, risks, and impediments during testing – it’s crucial to keep a record of your business model, the risks involved, and any potential obstacles. This includes documenting assumptions made, the experiments run, and the lessons learned. If you don’t, you may fall into the trap of convincing yourself that you predicted the outcome of a test, even if it came out differently.
Create a central repository for findings – have a wiki, dashboard, or similar tool to capture and articulate your findings. There may not be a single product that fits all needs, but a combination of tools like Notion, Airtable, and Zapier can be used to connect different parts and create MVPs.
Build a playbook that outlines the experiments that work best for you – some tests will be optimal in your industry and your specific operations. Create a playbook with examples, tools, and templates to help you run these experiments. This way you don’t start from scratch every time you want to run a test.
What are the ethical implications for running product tests on people? How is AI changing testing?
The new attitude is that testing should be done with people, not on people – this means that the testing process should involve co-creation with your B2B customers, uncovering their real needs together. This approach discourages the practice of tricking customers into buying something that cannot be built, which has been a prevalent issue in the past.
Avoid making false promises as part of a test – there has been a backlash against practices such as creating landing pages for products that will never be built or crowdfunding for projects that never see the light of day. This has led to a negative perception of the Lean Startup approach. The key is to ensure that your testing and development processes are transparent and honest.
Some companies are using AI products to replace human feedback – these AI customers provide feedback based on what they think you want to hear. While this may seem like a convenient solution, it’s important to remember the value of genuine human feedback in the development process.
Overall
What are the most important things to get right?
Maintain an open mind to ideas and testing – this allows you to be flexible and adaptable, which is essential in the ever-changing business landscape.
Check customer requests against your business strategy – always cross-check customer requests against your business strategy. This practice ensures that you’re not just building what the customer asked for, but also considering how it aligns with your overall business objectives.
Understand customer needs before building based on a request – dig deeper into the needs behind customer requests. This understanding allows you to develop solutions that truly address their problems, rather than just fulfilling their immediate requests.
Embrace being right about being wrong – embrace the possibility of being wrong. This approach allows you to learn from mistakes and make necessary adjustments, leading to better product ideas in the long run.
Prioritize progress over perfection – the goal of testing B2B product ideas is not to be right all the time, but to make progress. This mindset encourages continuous improvement and reduces the risk associated with your work.
What are common pitfalls?
Allowing insufficient time for testing – many businesses fall into the trap of not dedicating enough time to testing. This can lead to premature conclusions and decisions based on incomplete data. It’s crucial to allow ample time for testing to ensure accurate results.
Making large bets on light evidence – another common pitfall is making significant decisions based on light evidence. Businesses should avoid making large bets without substantial, reliable evidence to back them up.
Confirmation bias – this is a psychological phenomenon where we tend to favor information that confirms our preconceptions, regardless of whether the information is true. In a business context, this can lead to ignoring evidence that contradicts our worldview. It’s important to remain objective and open to all evidence, even if it challenges our initial beliefs.
Partial addressing issues – many B2B customers are only addressing about half of their issues effectively. This can lead to unresolved problems and missed opportunities for improvement. Businesses should strive to address all issues fully to ensure optimal performance.
Responses