Quebec’s agricultural industry is facing a growing structural challenge: seasonal labour shortages. In 2022, nearly 28,200 agricultural positions went unfilled across Canada during peak season, resulting in an estimated $3.5 billion in lost sales — and the fruit and vegetable sector is projected to be the hardest hit, with more than 35,000 peak-season positions to fill by 2030. For large-scale vegetable producers, post-harvest operations — handling, washing, labelling, and packaging — still rely heavily on teams that are difficult to recruit and retain.
It is in this context that Agmanic Vision, a Quebec-based specialist in AI-driven industrial vision and member of the MVTec Certified Integration Partner program, developed the vision software component of a fully automated post-harvest line for Les Fermes Émile Gariepy, a local squash farming operation — designed and deployed in partnership with system integrator Génik.
At the heart of the solution: MVTec HALCON, whose combined 3D Vision, calibration, and Deep Learning capabilities made it possible to automate a process inaccessible to conventional industrial vision approaches — incapable of handling the natural and unpredictable variability of an agricultural product — processing a squash every three seconds, across multiple varieties, shapes, and sizes.
The Challenge
Prior to automation, all post-harvest operations at Les Fermes Émile Gariepy were performed manually. Four operators per shift were required to load the washer, monitor the line, and pack squash into market-ready boxes — a recurring staffing challenge that reflects a broader national trend in which two out of five Canadian agricultural employers report being unable to fill all their seasonal positions.
The product itself made automation exceptionally demanding. Squash exhibits extreme natural variability: depending on the variety (butternut, acorn, spaghetti, and others), each unit presents a different size, shape, orientation, and weight. This heterogeneity immediately rendered traditional industrial vision systems inadequate — which require uniform, calibrated products and fixed detection parameters to function reliably. A platform capable of learning and generalizing across an open-ended range of shapes was essential.
The project imposed three distinct vision challenges to be solved simultaneously:
- Squash picking from harvest bin: Detect and locate squash randomly positioned inside a large wooden bin (48 x 48 inches), determine their precise 3D position and orientation, and guide a robot to grasp each one without colliding with the bin walls.
- Squash picking from washer output conveyor: Measure the size and position of each squash moving along the washer outlet conveyor, enabling the robot to grasp it on the fly and calculate the optimal deposit position to prevent product damage.
- Squash packing into boxes: Dynamically plan squash placement inside carton boxes in real time, accounting for units already deposited, to maximize fill and reach the target capacity per box.
Each challenge required a different vision capability — and all three had to operate together within a single, continuous production line.
The Solution
Génik designed the solution around two robotic cells — a Bin Picking Robot and a Bin Placing Robot — connected by a brush washer and an automatic labelling section. Agmanic Vision developed the entire vision software component, with MVTec HALCON as the unified platform for both cells, running on an industrial computer equipped with an NVIDIA GPU that handles all image processing and communicates results to the robot controllers in real time.
Three cameras handle all vision functions on the line: a Basler Blaze 101 (3D time-of-flight camera) mounted directly on the Bin Picking Robot tool, moving with the arm to scan the wooden bin and generate a dense Point Cloud of its contents; and two fixed active stereo 3D cameras on the Bin Placing side — one above the washer outlet conveyor to detect each squash in motion, the other above the active carton box to plan each deposit position.
How MVTec HALCON Solves Each Challenge
Challenge 1 — Bin Picking: 3D Point Cloud Processing + Hand-Eye Calibration + Instance Segmentation
The Bin Picking cell combines three HALCON capabilities to transform raw sensor data into precise robot commands.
At each cycle, the Basler Blaze 101 generates a dense Point Cloud of the wooden bin interior — a complete three-dimensional representation of the squash pile. In parallel, HALCON’s Deep Learning Instance Segmentation analyzes the 2D image from the same camera to individually locate and delineate each visible squash, regardless of variety, size, or orientation. It is this combination — 2D image for detection, Point Cloud for depth — that enables precise identification of where each squash is located in space.
From this data, HALCON determines the grasping candidates: for each detected squash, the system validates its position and orientation by analyzing the Point Cloud around it, ensuring it is accessible and that the robot can reach it without colliding with the bin walls or neighboring squash. The best candidate is selected, and HALCON’s Hand-Eye Calibration — which establishes the precise spatial relationship between the camera mounted on the arm and the robot’s coordinate frame — translates this position into grasping coordinates directly usable by the robot.
Challenge 2 & 3 — Bin Placing: Eye-to-Hand Calibration + 3D Point Cloud Processing + Planning Algorithm
The Bin Placing cell uses a different calibration strategy that reflects its fundamentally different geometry. Here, the two active stereo 3D cameras are fixed in the cell — they do not move with the robot. HALCON’s Eye-to-Hand Calibration establishes the spatial relationship between each fixed camera and the robot’s coordinate frame, enabling the system to convert image-space measurements directly into robot workspace coordinates with the precision required for on-the-fly grasping.
For detecting squash in motion on the conveyor, HALCON performs a conventional Point Cloud analysis from the active stereo 3D camera — determining the position and size of each squash with the latency required to enable a clean pick without stopping the conveyor.
Above the carton box, the second active stereo 3D camera generates a Point Cloud of the box interior at each cycle. HALCON analyzes this Point Cloud to identify the optimal position to deposit the next squash, accounting for units already present and the dimensions of the incoming squash.
Performance
The fully integrated system processes one squash every three seconds continuously — 1,200 squash per hour — across multiple varieties, without operator intervention and without line reconfiguration between varieties.
Overcoming Technical Challenges
Natural variability — why traditional vision systems were not an option
The core technical insight of this project is that squash cannot be treated as an industrial part. Template matching, color thresholding, and shape-based detection methods all assume a known, repeatable object geometry. Squash violates every one of these assumptions: the butternut is elongated and smooth, the acorn is round and deeply ribbed, the spaghetti is cylindrical — and within each variety, individual size and orientation vary continuously.
HALCON’s Deep Learning Instance Segmentation was selected because it operates on a fundamentally different principle: rather than matching against a fixed template, it learns the concept of “squash” from a representative training dataset and generalizes to new instances it has never seen. This is what allows a single deployed model to handle the full variety mix on the line without rule updates or variety-specific configuration.
On-the-fly grasping — precision under time pressure
Grasping a moving object on a conveyor requires the vision system to not only detect the squash but predict its position at the moment of robot contact — all within a latency budget tight enough to match the conveyor speed. HALCON’s real-time processing pipeline, running on the NVIDIA GPU-equipped industrial computer, delivers the detection and coordinate calculation within the timing constraints required for clean, repeatable grasping without conveyor stops.
Dual calibration strategy — one platform, two geometries
One of the key strengths of the vision component developed by Agmanic Vision is the use of HALCON to manage two fundamentally different camera-robot configurations within a single integrated system. Hand-Eye Calibration for the moving Bin Picking camera, and Eye-to-Hand Calibration for the fixed Bin Placing cameras — both handled natively within HALCON, with results communicated seamlessly to the respective robot controllers. This unified software approach reduced integration complexity and provided a consistent calibration framework across both cells.
Key Outcomes
The deployment of this MVTec HALCON-powered line has delivered significant operational results for Les Fermes Émile Gariepy:
- Reduced labour dependency: 4 operators per shift replaced on picking, washing, and packaging operations — a direct response to the structural seasonal labour shortage affecting Quebec’s vegetable farming sector.
- Sustained throughput: 1,200 squash per hour, continuously, across multiple varieties — a rate unachievable manually and beyond what conventional vision systems can deliver on products with high geometric variability.
- Variety-agnostic operation: HALCON’s Instance Segmentation handles the full squash variety mix on a single line, without reconfiguration between varieties — a capability that conventional industrial vision systems cannot match.
- Optimized box filling: the planning algorithm developed by Agmanic Vision using HALCON functions maximizes carton box occupancy at every cycle, consistently reaching the target capacity per box.
“The natural variability of squash is exactly the type of challenge that conventional industrial vision systems cannot overcome. By combining Point Cloud Processing, Instance Segmentation, and the calibration tools of MVTec HALCON, we were able to automate a complete line on a non-uniform agricultural product — and reach 1,200 units per hour.”
— Peter McLaughlin, Managing Director and Co-Founder, Agmanic