Market to Plate: The Whole Pipeline

The lesson

Half past 4, one bulb burning

It is 4:30 in the morning and most of the stalls in the hawker centre are shut. Mine is the only one that has its lights on, and I am writing the day’s market list. Behind me, the wok is cold. There is no fire at this hour. The famous part of my day is frying the noodles in the wok over a live flame. That frying is what everyone remembers and it is the fastest part of preparing a plate of Hokkien Mee. Everything before it is tireless work you never see, under market lights and over a sink, and it takes up most of the day. Follow my whole day, and you would have walked through what every AI project goes through. There are 7 steps, and one plate.

Decide the dish before you shop

Step one happened decades ago, when I chose the problem to solve. I wanted to sell Fried Hokkien Mee for a living. That choice rules everything after it. An AI project starts the same way, by defining its problem. Step two is the wet market. That is collecting the data. Step three is the sink. Every prawn gets washed, peeled, and deveined. Cleaning data is the same joyless job, because the world sells gritty prawns and messy records. Step four is the stock in my pot. Hours of quiet boiling turn meat bones and prawn heads and shells into a rich stock. This is what feature engineering is. You boil raw data down to the good stuff a model can taste. Step five is picking the Hokkien Mee recipe. That is choosing the algorithm. Step six is frying the noodles in the wok. That is training, where the machine learns from examples and adjusts itself. Step seven is taste and adjust. That is evaluation, spoon by spoon.

Then I serve the plate of Hokkien Mee to the customer, and I keep listening. A model that goes live is watched the same way. Customers drift. A model that was right in January can be wrong by June. The listening helps me tweak and adjust my recipe and we go straight back to step one. That is why engineers draw the whole thing as a circle.

The same 7 steps at a bank

Run the same 7 steps at a bank. The only change is the uniform. Decide the problem: which card payments are fraud. Collect: years of transactions. Clean: repair the missing and mislabelled records. Features: boil them down to what matters, such as the hour and the amount. Algorithm: choose the sorter. Train: let it learn. Evaluate: test it on payments it has never seen, and see if it managed to predict the fraud.

Say the 7 steps, no notes

Now you walk the 7 steps yourself. Say the 7 steps to yourself. I give you three seconds.

Pause here for three seconds.

6 out of 7 beats most project meetings. Now the question that matters. Across the whole day, where did the hours go? Three seconds.

Pause here for three seconds.

Most of the day went into the hours before I fire up the wok. Preparation is the real work, and the wok is the smallest part of it. Every AI project runs the same way. The data work consumes the most time. The glamorous firing of the model takes a few minutes. Most projects are not lost at training. They are lost at the sink.