My Thesis - Tartar Tech

The data capture system

A few years ago, I finished my PhD. Unfortunately, my thesis wasn’t publicly available due to it’s commercial sensitivity. But now, it’s free for anyone to read. You can do so at The University of Waikato Research Commons, or download the PDF directly here.

To give you a taste of the juicy nectar it contains, here’s a very brief overview:

I developed a yield estimation system for kiwifruit.
Cameras (x4), a lidar and an IMU were mounted on a quad bike (ATV).
I rode it through a lot of orchards a few months before harvesting.
A CNN was used to detect fruit in the images, then stereo vision to locate them in 3D space relative to the vehicle.
The lidar and IMU were used to perform SLAM to locate the vehicle relative to the orchard.
You end up seeing the same fruit from multiple perspectives, so I developed a cool little algorithm to identify duplicate fruit.
Many fruit hide behind leaves, branches and other things, so I compared how many fruit I saw, to how many were actually harvested from orchards.
I then came up with a model that predicted how many fruit weren’t seen, based on characteristics of the orchard, such as leaf density, row width, fruit clustering and fruit distribution across the rows.
The result was a system that predicted actual fruit yield with a mean average error of 0.68% across 3 test orchards (test orchards were not used for development of the system).
Humans routinely have errors of over 20% when they manually count fruit (which is how it’s done in industry).
Also generates heatmaps (or other visualisations) showing the locations of all the fruit in an orchard.

The following images shows kiwifruit identified by the neural network I trained. The fruit are outlined in pink and the calyx (fuzzy bit on the bottom) of the fruit are shown in the white squares. Detected kiwifruit

Here is a visualisation of all the kiwifruit in one orchard. Each dot is a single kiwifruit and it’s colour represents the height of the fruit off the ground. The smaller areas without fruit are where the male plants are (only the females bear fruit). The larger areas are where plants have been cut out due to disease or other issues and new plants are not yet bearing fruit. The strip of low fruit (red in colour) in the middle of the orchard is where two previously separate orchard blocks have been joined together.

Here’s the full abstract of the thesis:

Kiwifruit is very important to the New Zealand economy, bringing in NZ$2.39 billion in the 2017/2018 season and projected to grow to NZ$4.5 billion by 2025. Every season, the expected yield from each orchard is estimated. These estimates are used for both on-orchard decisions regarding thinning and spraying, and industry level decisions regarding export pricing, supply agreements, labour requirements, packaging quantities and more. The current method for yield estimation is manual and prone to errors. In the 2016/17 growing season, green kiwifruit estimates were 5.9% lower than actual harvest yield, resulting in over 5 million trays (approximately 18,000 tons) of fruit being disposed of.

An automated system could reduce errors saving the industry significant amounts of money. Such a system could realise other benefits such as providing more granular information on orchard performance to growers and increasing labour efficiency and crop uniformity via targeted labour. Other automated fruit yield estimation systems have been developed but have suffered from sub-optimal performance due to occlusion of fruit. The work presented here introduces a system that has been developed with the aim of minimising occlusion where possible and employs an occlusion prediction model to account for remaining occlusion. The system utilises stereo vision, a 3D lidar unit, an inertial measurement unit and LED lighting mounted on an all terrain vehicle (ATV). Data is captured across 16 maturity areas, consisting of 33.9 Ha of commercial kiwifruit orchards. A simultaneous localisation and mapping system is used to locate the ATV within the orchard. A convolutional neural network detects kiwifruit, which are localised via stereo triangulation. Individual fruit are tracked through overlapping images to reduce occlusion while ensuring no double counting. Fruit are counted and an occlusion prediction model is applied to account for the unseen fruit.

The system is able to predict orchard scale fruit yield with a mean average percentage error of 0.68%. This error rate was obtained from an independent test dataset and ground truth data obtained from a commercial kiwifruit packhouse. The system can also generate visualisations of orchards showing all individual detected fruit in 3D, fruit density maps and canopy density maps. The ability of the system to predict yield with high accuracy and at orchard scale sets it apart from previous work. It is also the largest scale study currently known, consisting of over 15 million fruit.