![\"\"\/](\"https:\/\/images.unsplash.com\/photo-1517824806704-9040b037703b?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1950&q=80\")
You will<\/strong> end up building useless stuff. In a highly-uncertain startup setting you\u2019re guaranteed to write lots of code that you\u2019re going to need to throw away. <\/p>\n\n\n\n This is ok, because it\u2019s by writing this code that you learn which bits are useful and which bits are not. Some models won\u2019t work; others will work, but not provide business value; others will be superseded by some amazing open source implementation a month after you\u2019ve finished writing them. Your job is to make sure that the code you now need to throw away was written as quickly as possible.<\/p>\n\n\n\n There are lots of reasons why developers and scientists are tempted into overengineering. Here’s a few:<\/p>\n\n\n\n These are all bad<\/strong> reasons. It takes a lot of self-awareness to acknowledge that you don\u2019t understand the problem space well enough to justify building tooling. Until you really understand what you\u2019re solving, and the painpoints have become festering sores, you shouldn\u2019t be building \u2018my distributed framework for X\u2019 or \u2018automated dispatching of Y\u2019. <\/p>\n\n\n\n This problem is exacerbated in ML, because often the loop time is long – it can take a long time to train a model. Going through the iterations needed to figure out what is useful or not can become very time consuming. This is further exacerbated if you don\u2019t have labelled data, in which case you\u2019re also going to have delays whilst data gets labelled. The solution to both of these is to try and stick to very simple models, trained on very small datasets. This will help you keep your loop time manageable until you\u2019re certain<\/em> you\u2019re solving a real problem.<\/p>\n\n\n\n Most of the code you write in an applied research context is not about adding a feature; it\u2019s about testing an idea. You need your code to be good enough that you can trust the conclusions that you\u2019re going to draw from it. If you screw up your evaluation pipeline, for instance, you\u2019re in all kinds of trouble, because you won\u2019t be able to tell whether the changes you\u2019re making to the codebase are actually making things better or worse.<\/p>\n\n\n\n But by and large, it\u2019s not worth writing your machine learning code in the expectation that it will be reused. Code designed to be easily extensible is often harder to read and more prone to bugs than simpler implementations. Yes, you may find that building 6 layers of inheritance from your DeepCNN base class prevents you copying and pasting some code at some point in the future. But it\u2019s more likely that you find out that actually you should be using an RNN, and you end up throwing away the CNN entirely. Don\u2019t take my word for it: Andrej Karpathy agrees.<\/a><\/p>\n\n\n\n Sometimes it\u2019s tempting to put aside your reservations and embark on building something you\u2019re not really sure will be helpful, but somebody else<\/em> thinks is important. Be very wary.<\/strong> You and the team will eventually need to decide whether to maintain this feature or whether to kill it. Killing things is demoralizing, particularly when everybody involved acknowledges that it didn\u2019t need to be built in the first place. If there\u2019s even an inkling of doubt: You Ain’t Gonna Need it (YAGNI).<\/p>\n\n\n\n Building seemingly useful stuff is seductive, and so is learning seemingly useful stuff.<\/p>\n\n\n\n Generally, the best way to learn the set of skills you need to build cool things is to go ahead and build cool things. There may be specific instances where it\u2019s more efficient to do some tutorials than to dive straight in, but if you want your team to get really good at building and debugging deep learning models, they should be doing that. The space of things you could learn that would not help your company build great products is literally infinite. <\/p>\n\n\n\n There\u2019s an awful lot of novelty in software and especially <\/em>ML, and it requires discipline to resist the FOMO associated with the release of each new pipeline tool, framework, or pretrained model library. If it solves a problem you are suffering from, then maybe it\u2019s worth learning. But it\u2019s worth learning because it solves a problem, not because \u2018learning\u2019 is inherently good. <\/p>\n\n\n\n I think this is the right direction to lean in, but this can lead to a bit of tunnel-vision. Correctly managed, giving the team some fraction of their time to explore problems outside of their immediate scope – Google advocates 20% – can be very impactful. But I think it\u2019s necessary to ground that exploration in real problems, rather than random wanderings in machine-learning land. <\/p>\n\n\n\n The challenge of research is that it\u2019s often really hard to predict the return on effort invested.<\/p>\n\n\n\n This is what the US Marines refer to as a \u2019non-linearity\u2019: a small investment in the right place at the right time can produce outlandish results. For instance, a minor change in your preprocessing pipeline might reveal a bug that causes a massive drop in performance; adding a pretrained embedding can take your optimization problem from intractable to converging; a new open-source implementation might solve your problem straight out of the box.<\/p>\n\n\n\n Because you cannot know in advance which particular investment will pay off in droves, you need to be able to keep your research agenda fairly diverse. I\u2019ve found it common for teams to double down upon too small a subset of ideas. It feels good if everybody in the team is pulling in the same direction, and communal understanding of these ideas is progressively deepening and maturing. But inevitably inertia and sunk-costs lead you to overinvest in your current agenda. It\u2019s easy to get stuck in a local minima where you see drastically decreasing returns with increasing investment of effort.<\/p>\n\n\n\n The converse is not much better: everybody working on completely disparate approaches to a problem usually leads to isolation, bugs, and completely unintelligible code. This is what you often see in academic science. Because it\u2019s conducted by lone individuals, the research artefacts produced are often of very poor quality; without constant scrutiny from peers, it\u2019s difficult to spot errors. Furthermore, the majority of careers in science fail, because the ideas that were being pursued didn\u2019t work. This isn\u2019t a dynamic you want to encourage in your teams.<\/p>\n\n\n\n I think the right decision here is a function of team age. With a young team, having everybody working closely together allows the mutual oversight that junior practitioners benefit from, as well as accelerating the development of camaraderie and team unity. Conversely, as the team and the individuals within it tend towards maturity, you want to allow more divergent agendas, giving people the space to follow their hunches without necessarily justifying every single line of code to the team as a whole. You need to be able to place a lot of bets, and the best way to do that is to allow people to explore their own ideas in individuals or pairs rather than straitjacketing them with a stifling planning process.<\/p>\n\n\n\n When doing work with clients or other teams, you will rarely regret investing time in defining and writing down scope.<\/p>\n\n\n\n I\u2019ve been surprised at how reluctant people are to locking down written scopes for pieces of ML work. In part, it comes from the popularity of \u2018Agile\u2019 methodologies, which encourage you to build fast and iterate later. Whilst these ideas have definite value, my experience suggests that failure to scope at the start of a piece of work usually means you end up scoping it halfway through, once expectations around timelines and deliverables have already started to crystallize – and become resistant to change. This upsets the client and your team, both of whom come away with the impression that nobody really knows what\u2019s going on. The line between \u2018Agile\u2019 and \u2018poorly thought through\u2019 is a fine one.<\/p>\n\n\n\n The most useful feature of a really good technical scoping is that it highlights your assumptions. In the case of machine learning, those assumptions are often about the data. <\/p>\n\n\n\n Given the generally deplorable state of enterprise data, it\u2019s worth pushing to make sure the inputs and outputs to the system that you\u2019re building are completely understood by everybody – and that they match the data actually available in whatever system you\u2019re planning to integrate into. Even small misunderstandings about what data is stored and where, or why<\/em> certain pieces of data get stored and others do not can lead to the specification of systems that are untrainable (because the requisite data was never stored), un-integratable (because what they produce doesn\u2019t match what the current system produces) or biased (because they\u2019re trained on a weird subset of the data).<\/p>\n\n\n\n If you’re not technical, you have<\/em> to have deeply technical ML people involved in this process. They might embarrass you by asking what a KPI or a POC stands for in front of a client, but you’ll be bloody glad when they point out that the data you’re asking them to train their models on doesn’t actually exist (yes, this happens).<\/p>\n\n\n\n Machine learning scientists are overly-quantitative.<\/p>\n\n\n\n Quantitative is a lovely word, associated with serious people who make lots of money and conduct their research in rigorous ways without allowing opinions or judgements to creep in.<\/p>\n\n\n\n The problem is that humans – even scientists- just aren\u2019t really very good at interpreting numbers<\/a>. Furthermore, the numbers we use in machine learning often defy intuition (my loss is 0.90, anybody?). An over-reliance on numbers can often hide bugs in your code: printed predictions rarely lie, but wiggly loss curves often do (more war stories in this post<\/a>). <\/p>\n\n\n\n The problem is so much worse when you try to communicate how your model is performing to potential users. Those \u2018opinions\u2019 and \u2018judgements\u2019 we wanted to avoid creeping in are what people will rely upon to decide whether to buy your product. If you give them numbers they don\u2019t understand, they will form opinions and judgements of those instead.<\/p>\n\n\n\n2. Not all learning is equal<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/i0.wp.com\/images.unsplash.com\/reserve\/oIpwxeeSPy1cnwYpqJ1w_Dufer%20Collateral%20test.jpg?w=660&ssl=1\")
3. Place lots of bets, but keep the team together<\/h2>\n\n\n\n
4. Time spent scoping is never wasted<\/h2>\n\n\n\n
5. Success is usually qualitative, not quantitative<\/h2>\n\n\n\n
![\"\"\/](\"https:\/\/images.unsplash.com\/photo-1518082903679-22ec3732ff0c?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1950&q=80\")
![\"\"\/](\"https:\/\/images.unsplash.com\/photo-1496950866446-3253e1470e8e?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1950&q=80\")
![\"\"\/](\"https:\/\/images.unsplash.com\/photo-1507777163158-e166827ed01c?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=2700&q=80\")