Hypothesis Based Learning

 

In my experience, the most sophisticated decision makers tend to be hypothesis-driven thinkers. They may be engineers solving a technical problem, product designers fulfilling a customer need, or entrepreneurs growing a business. They form a hypothesis about how to reach their goal and then work systematically to either validate or falsify it. 

 

Say you’re tuning a learning algorithm that estimates the health of corn stalks based on input from a tractor-mounted camera. (Many companies are developing products like this to help farmers make decisions about planting, weeding, or harvesting.) If your algorithm is doing poorly, how should you go about improving it? 

 

Some engineers tend to apply a one-size-fits-all rule. Someone who has experience improving algorithms by collecting more data may tend to gather more photos of corn stalks. When that doesn’t work, they may end up trying things more or less at random until they stumble on something that works.

 

Hypothesis-driven thinkers, on the other hand, have seen learning algorithms perform poorly for many different reasons. Based on that experience, they can make a list of hypotheses about what could be going wrong. Perhaps the algorithm does well in sunlight but performs poorly on overcast days, in which case the best solution indeed may be to collect — or synthesize — more images under cloudy skies. Or perhaps the camera’s lens is obscured by dust, or the hyperparameters are poorly tuned. 

Hypothesis-driven thinkers see a variety of possibilities. They pick the most likely one and carry out error analysis or other tests to falsify or validate it. Them they apply the insights they've gained to devise a solution, choose a new hypothesis, or re-evaluate the range of hypotheses. In this way, they find a good solution efficiently.

 

How can you gain skill in building hypotheses?

• Seek out stories of how others have built machine learning systems. Learning from friends and colleagues about not only what worked but also what path led there — including wrong turns and ideas considered and rejected — can hone your intuition. 
• If you work with other engineers and they advocate a course of action, ask why. Conversely, if you favor a particular approach, share your reasoning and invite them to challenge you. This helps you to (i) gain exposure to more tactics and (ii) understand when various tactics apply. 
• Keep taking courses. They can expose you quickly to a wide range of examples. 

Hypothesis-driven thinking is helpful not only in developing AI systems, but also in building products and businesses. Perhaps you’ve identified a market need, a concept to fulfill that need, and a sales strategy to get the product into the customers’ hands. Rather than rushing ahead and assuming that everything will work out, you might question key assumptions behind the hypothesis systematically and pinpoint those that are unproven or incorrect. If you discover early on that the concept is flawed (say, because it requires AI technology that hasn’t been invented yet), you’ll have more time to pivot and find an alternative. 

Source: Andrew Ng