Classifying fish - a machine learning approach

Neil Dhir October 2, 2019

Fish. Everyone likes fish. At the very least, at Kamin, everyone likes fish. As it turns out you can combine the art of fishing with machine learning and computer vision.

So what was the problem? Well, the customer in question, a large Swedish distributor of said aquatic animal, was having a somewhat troubling time separating one species from another, on their conveyor belt. The old solution simply involved lots of people sorting all of that fish by hand. But when you are dealing with several tonnes of fish, that simply will not do, so they needed another solution; automatic fish classification (using machine learning and computer vision).

A herring fish
Figure: What am I (NB: fish have existential crises too)?


Imagine now that you are in a great big factory, and there are loud noises, people buzzing about, and you are trying to figure what exactly is coming at you (on that conveyor belt) at a rather alarming rate. In our case there were three things the customer wanted to know, and this is what formed the basis for our classification task:

  • Is the fish a herring or a mackerel (or indeed, is it a fish at all)?
  • Once you've figured out it is actually a fish (and not a sea-snake say), determine if the fish is in the right or wrong orientation.
  • Finally, having done the above; determine if the fish is damaged or not.

Fish images

That is a rather interesting problem you might think (and you would be right). Consequently, the first thing you should be asking yourself now is: what's the data? It is good that that was your first question as well, because that is certainly where we started. Our customer provided us with images like the one you see below, whereupon our model has superimposed a bounding box to indicate it has identified a herring.

Figure: Lots of fish. Hard to tell which is a mackerel.

Now, we got a few of these images, lots actually. We got a bunch for each of the cases that we had been asked to identify. Once we were satisfied with the data, we got working on the actual model.


You might not believe it, but rarely do you find yourself in your professional life, asking what model should I use to classify my fish? But lo and behold; that is where Kamin's employees found themselves one day. Luckily they knew exactly what tools to adopt for this interesting problem. Before we move onto the juicy bits you may wonder why we didn't write a set of rules and heuristics to capture the variations in appearance, colour and size of the fishes. The answer is simply that it is virtually impossible. The amount of variation present, even in a problem as simple as this, is staggering, hence any rules written would be quickly overwhelmed by edge-cases not previously considered. Thus enter machine learning.

Figure: Imagine that the cat is a herring and the dog a mackerel.

If you haven't been living in a cave for the past few years, you'll know that image analysis and image classification (which is what is shown in the above image) has vastly improved, especially with the introduction of convolutional neural networks (CNN) – commonly used with deep learning (DL). CNNs are a specialised form of DL where a large number of filtering and sampling operations are connected in a network. The network learns to classify images by optimising the model's parameters using a set of training images and recognises similar feature therein, much like a human would. The CNN models developed take any standard image format as an input and returns the probability of the image belonging to each of the classes of interest to us. A fish image is analysed in 10ms and classified with an accuracy and precision above 99.9%. As it stands, CNN models are considered state-of-the-art in terms of accuracy and robustness, however, they do require larger amounts of training data compared to traditional methods. Luckily for us, we did have a lot of data in this project so we were set for success.

In sum

This was a very interesting and exciting project. Not only did it demonstrate where simple machine learning and computer vision methods could be applied, it also highlighted the ease with which some processes can be automated. A derivative of this work is currently ongoing with another customer, who has a much more challenging fish classification problem.

You might hear more about that another time...