Learning to perfect machine vision

 AUTONOMOUS AGRIBOTS: With the right technology a simple UAV like this one could scan a field or orchard, plant by plant, gathering data to help growers make the best decisions for maximizing yields while minimizing costs.

Imagine a future in which comprehensive data about maturing crops is collected, analyzed and delivered to growers in real time. That information that could be used to help make the best decisions about irrigation, pesticide and fertilizer application, optimal times to plant and reap, and to accurately estimate crop yields days or weeks before harvest. It could also aid researchers in developing new growing methods and tools.

All this could revolutionize agriculture just as the tractor did, saving untold billions in labour, inputs and crop loss, thereby making food cheaper and easier to produce. It’s an approach called precision agriculture and it’s something governments and researchers enthusiastically support. But growers can’t realistically watch and record real-time data on every plant around the clock. They’ll need some new technology.  

“One way is to send an autonomous robot into the field to collect data, analyze the crop and report back,” says Queen’s engineering instructor and researcher, Tahir Ahmed.

One of the most difficult parts of building such a tool is machine vision: giving robots the capability to see, understand and make decisions about what’s happening around them. Ahmed has specialized in that technology for the past 10 years and is an instructor of ELEC474, Machine Vision. He’s a reasonably new teacher – this is his second course – and after gaining some industry experience and earning his PhD, he’s planning for a career in academia. He took an innovative approach to the exam for ELEC474 last term.  

PRACTICAL APPLICATION: Queen’s engineering instructor Tahir Ahmed challenged students to solve a complex machine vision problem in short order. “I was impressed, really impressed,” he says.

“The technology isn’t done,” he says. “People are still investigating but I gave students a very small part of this problem: A robot has captured video of an apple tree and its job is to count the number of apples on the tree in real time. When the camera moves, the count is maintained.”

The problem is more difficult than it sounds. Students were given only a piece of video from an apple orchard in which the camera moves in a slow arc around a tree. Not all the apples are the same size, shape or colour. As the camera moves, apples move into and out of frame, the views of others are blocked by other apples or leaves. Still, the assignment is to write code that keeps as accurate a count as possible in real time of how many apples there are on the tree.

“I wanted them to apply what they learned in the course,” says Ahmed. “This was a realistic problem they had to solve by thinking on it and using newly learned techniques to come up with an application. The most difficult part was the timeframe. They had to complete the assignment within one-and-a-half days and they did a really great job; way better than my expectations.”

ELEC474 student Alex Wiseman developed an elegant solution Ahmed’s to challenge.

“It was pretty difficult because the apples weren’t uniquely coloured,” says Wiseman. “Because of the way the light fell, different shades made it difficult to segment the apples and detect them.”

ELEGANT SOLUTION: Queen’s ECE 4+1 student Alex Wiseman earned top marks for her work on the ELEC474 exam problem. Now she’s moving on to apply her skills to research involving virtual reality.

Wiseman wrote code that used a colour segmentation technique she learned in one of the course labs to count the apples. She broke each video frame into colour channels, then thresholded each channel to distinguish individual apples. Once the apples were identified and counted, she compared each frame to its neighbours to establish which apples were newly identified for counting, which had already been counted and which were moving from view.

“I started with one frame of the video and I just wanted to get the apples detected in one frame and ignore the problem of detecting all the apples in the whole video,” says Wiseman. “Once I had something working, I’d build upon it and get the next frame working and so-on. It was very tough. It took a number of hours and was much more challenging than anything else we tried in the course up until then.”

Wiseman got top marks for her solution and it’s not surprising considering her interest in machine vision. She’s an ECE 4 + 1 student and is studying similar ideas in pursuit of her master’s.

“I’ve always liked any kind of artificial intelligence and machine vision is a subset of that,” she says. “We’re working on localization in mines, so if you don’t have a GPS signal you can figure out where you are just by using images. We’re porting that to an augmented reality headset.”