Soft Robotic End-Effectors in the Wild: A Case Study of a Soft Manipulator for Green Bell Pepper Harvesting
Abstract
In this case study we present a proof of concept design of a soft robotic end-effector for green pepper harvesting.
We test our robotic system in a standard double-row cropping system under real-world conditions without making modifications to the pepper plants or the environment.
We show that soft manipulators can reliably grasp peppers without bruising the fruit and provide an in-depth analysis of common types of failures we observe.
Finally, we discuss future improvements to our robotic system.
Introduction
With the agricultural sector undergoing a technological transformation largely fueled by global labor shortages the demand for automated harvesting is becoming more widespread.
Recent progress in robotic harvesting can be observed amongst most commonly cultivated crops such as cherry tomatoes (Feng et al. 2018), apples (Silwal et al.
and sweet peppers (Lehnert et al. 2017; Arad et al.
However, most approaches are geared towards largescale commercial operations and were both developed for and tested in controlled environments such as greenhouses, vertical farms and orchards.
This trend is especially problematic for small or mid-sized farms which are much more restricted to labor-intensive methods of harvesting their produce due to the cost-prohibitive nature of automation.
In this paper we therefore study the requirements for a fully automated robotic green pepper harvester operating in a standard commercial double crop row system; the most common form of growing sweet peppers in North America on small and mid-size farms.
In our tests we do not make any modifications to plant or environment.
We identify challenges related to grasping and cutting fruits and provide a proof of concept design for a low-cost soft robotic endeffector and cutter for green pepper harvesting.
Further, we report findings of operating a teleoperated fully robotic system in the field under real-world conditions (fig. 1) and suggest detailed improvements to the current system.
Related Works
Robotic harvesting of green peppers poses a multi-faceted challenge to research due to the unstructured environment and real-world operating conditions.
Thus, previous works typically focus on solving sub-tasks such as detecting peppers (Vitzrabin and Edan 2016; Ostovar, Ringdahl, and Hellstr ¨ om 2018; Zhao et al. 2020;
Ning et al. 2022), scheduling and selecting peppers for harvest (Zion et al. 2014), and grasp or motion planning for picking the peppers (Bac et al.
2016; Ringdahl, Kurtser, and Edan 2019; Kurtser and Edan 2020).
One of the first robotic systems for pepper harvesting in greenhouses was introduced by (Kitamura and Oka 2005).
More recent works by (Bac et al. 2017; Lehnert et al. 2017), and (Arad et al. 2020) have significantly improved the stateof-the-art but are still far below what is commercially viable in terms of harvesting accuracy and speed.
Most works rely on a custom end-effector design consisting of a gripper and a cutting device that is mounted to a robot arm.
(Kitamura and Oka 2005) use a parallel linkage mechanism and a pruner to cut the pepper stem.
(Hemming et al. 2014) developed two different types of end-effectors, one fin-ray based gripper with standard scissors and a suction cup based gripper with a hinged jaw mechanism for cutting.
Another suction cup based system was developed by (Lehnert et al. 2017) which works in combination with an oscillating blade.
(Arad et al. 2020) also use an oscillating blade to cut the peduncle but rely on a passive fruit-catching device to hold the fruit once cut.
Test Environment and Design Requirements
Our long-term goal is to operate a fully automated robotic harvester in an open field without any additional modifications to plants or fields.
In this case study we identify the main challenges associated with detection, grasping, cutting, and removing green peppers from the plant.
Accordingly, we choose to test our robot in a typical standard double-row cropping system with 12-inch spacing between plants and 6ft spacing between rows on an open field as shown in fig.
We aim for the following robot capabilities:
• Ability to achieve stable grasps of the fruit without damage to the fruit itself or other fruits • No damage to the plant stem or leaves • Clean cut of the pepper stem without twisting or damaging the peduncle (stalk supporting an individual fruit)
Printable Soft End-Effectors for Bell Pepper Harvesting
In our previous work (Bauer et al. 2021) we hypothesize that soft end-effectors can intrinsically provide the required compliance for handling peppers without bruising the fruit, or causing damage to the plant.
To verify this claim, we create a proof-of-concept design of a fully printed low-cost endeffector following the design and fabrication methodology outlined in (Bauer et al. 2022).
In addition to gently grasping the fruit, detaching the fruit from the peduncle is another critical step in the harvesting process.
In this section we present our proof-of-concept design.
More images of the geometric features of the end-effector can be found in the appendix.
The three detachable fingers and cutting shears are driven by servo motors enclosed in the wrist via tendons.
End-Effector Design
The end-effector design is shown in fig.
It features three fingers that are printed using TPU filaments (NinjaTek Chinchilla).
Each finger is detachable and can be customized to different crop varieties by changing the width and length of the finger.
To determine the finger dimensions for our proofof-concept design we choose the length of the fingers such that the end-effector can grasp a fully grown green pepper.
Each finger is actuated with a tendon that is driven by a servo motor (Dynamixel XC330).
Cutting Mechanism Design
Most current robotic harvesting systems feature a fixed blade design that is attached directly above the end-effector (Arad et al. 2020; Lehnert et al. 2017; Bac et al. 2017).
While practical, this design choice severely limits the robot’s ability to cut difficult-to-reach peduncles without cutting either fruit or plant stem.
This, in part, is reflected in the rather low success rates of state-of-the-art systems.
In contrast, humans use both hands for harvesting peppers; one hand grasps the fruit while the other hand is cutting the peduncle.
This allows humans to detach fruits even if they are growing close together or next to a stem because they can easily approach and cut from a variety of different directions.
A similar solution would require the addition of a second robotic arm which would significantly increase the upfront costs of the robot.
Thus, we choose to attach pruning scissors to a short, flexible arm that protrudes directly from the motor casing of the end-effector as shown in fig.
This allows the robot to achieve a wide variety of cut directions relative to the grasping direction.
We actuate the pruning scissors through a tendon-pulley system.
For this case study we do not actively control the cutting arm.
Instead, our goal is to verify that an independently movable cutting mechanism is beneficial for harvesting peppers in an unmodified field.
System Design and Control
The full robotic system is built on a Husky mobile robot base (Clearpath Robotics Husky) to which we attach a UFACTORY X-ARM 6.
The system is battery-powered and can be operated for more than 4 hours.
We mount the custom end-effector directly to the X-arm via M6 screws and supply 12V DC power from the battery located in the Husky robot.
Communication with the end-effector is established over a USB connection.
Robot communication is based on the Robot Operating System (ROS) and the robot can be teleoperated using a standard gamepad.
There are two main teleoperation control modes; In-field driving and arm control.
The arm control mode is used to teleoperate the robot arm and end-effector in cartesian space.
In this mode we map joystick commands to cartesian end-effector velocities by using a built-in cartesian planner from the X-Arm.
The end-effector is operated via button pushes, where the fingers and the cutter move into pre-defined poses using open-loop control.
Experiments
In-Hand Operation To evaluate whether the custom soft end-effector is able to operate within an open field in accordance with the requirements formulated above, we conduct our first experiment by placing the end-effector by hand.
We manually bring the gripper into a pre-grasp position and actuate the fingers to grasp the pepper.
The blades of the cutter are manually positioned around the fruit stem.
The stem is automatically cut, we can retract the gripper holding the pepper, and finally open the gripper to release the fruit.
Teleoperated Full Robotic Platform To demonstrate our gripper’s capabilities in a typical pepper growing setup, we deploy the fully robotic system (Husky mobile base, X-Arm, and gripper) between rows in a standard double-row pepper cropping field, as shown in fig.
We drive the robot next to the plant and then switch to arm control mode for harvesting the pepper using the gamepad controller.
Figureshows a successful harvesting attempt from left to right: The robot approaches a pepper plant (A), moves into a pre-grasp position (B), and closes the gripper (C).
We manually position the cutting mechanism at the fruit stem (D), and trigger cutting using the wireless controller (E).
Finally, the arm is retracted to retrieve the pepper (F).
Results
In our full robot field test, we observe that grasping attempts are successful in most cases.
Overall, we harvested a total of 32 peppers, none of which were bruised by the soft fingers (as shown in fig.
4-C) even though the end-effector control was implemented as simple open-loop control.
Especially for ripe full-size peppers that are growing unobstructed, we were always able to achieve a stable grasp as shown in fig.
4B. Additionally, we did not observe any visual damage to plants or foliage and the pruning blades resulted in clean cuts as shown in fig.
4-A. This can largely be attested to the soft and compliant nature of the end-effector and is a strong indicator that using soft hands in agriculture applications could be a viable low-cost option.
However, we identify a number of plant configurations that make grasping and cutting difficult and lead to failures.
Generally, we observe two failure modes: grasp and cutter failures.
These can be further sub-classified as shown in fig.
Most grasp failures result from direct obstructions of the fruit body by either the plant stem (fig.
5-B), another fruit growing adjacently (fig.
5-C,D) or foliage preventing the fingers to tightly close around the pepper (fig.
5-A).
Another reason for unstable grasps are a poor approach direction and the fruit slipping through the fingers due to the pepper being too small.
Even though we teleoperated the robot in our experiments we were not always able to find an optimal approach direction to grasp the pepper.
In some cases, the husky position relative to the plant combined with the workspace of the six degree of freedom (DOF)
X-Arm created a very narrow window of possible approach angles.
Since we tested our prototype in late July, some of the peppers were too small for our gripper to grasp; we designed the end-effector to harvest fully grown fruits.
This failure is not a problem with the gripper design, but rather with harvesting the peppers too early.
Since we manually placed the pruning blades in this case study, we did not observe many failures when detaching the fruit.
In rare instances of cutter failures we were not able to place the cutter properly either because the peduncle was nestled against the fruit completely, or the peduncle grew right along the plant stem.
Apart from this, we found that an independently movable cutter allows us to reliably and safely cut the peduncles; this suggests that a fully actuated version of our cutting device could improve the overall success rate of current systems.
Suggested Improvements Detection
Compared to colored peppers, green peppers are difficult to detect between the foliage.
Based on our experimental re- sults we find that peppers can best be seen from a lower view that is slightly angled upwards towards the canopy.
Our suggestion is to therefore place a camera on the side of the husky base.
We propose the use of an RGB-D camera, in combination with a single shot detector such as YOLO-V similar to the work by (Ning et al. 2022) who have shown that this approach can achieve a high detection rate under varying lighting conditions.
In order to expose hidden fruits to the camera view we also suggest the use of low-pressure air to blow against the foliage.
A second camera for visual servoing could be placed directly in the end-effector palm.
Grasp Planning
In order to grasp peppers autonomously, a strategy is required that successfully positions the end-effector into a pregrasp configuration.
A commonly used strategy in harvesting is visual servoing (Han et al.
Dewi et al.
Arad et al. 2020).
However, visual servoing alone will often not be successful due to the many environmental obstacles and workspace restrictions of the robot.
Therefore the robot will need to evaluate a variety of different approach trajectories, similar to the strategies outlined by (Ringdahl, Kurtser, and Edan 2019), and use its cameras to build a 3D model of the plants as outlined by (Yandun, Silwal, and Kantor 2020).
Cutter Design
A freely movable cutting device proved to be beneficial for cutting difficult-to-reach or occluded peduncles.
While the pruning scissors achieve very clean cuts, placing the blades between the peduncle correctly in a pre-cut position will require the robot to have a precise estimate of the peduncle location and orientation.
To create a low-cost durable fully actuated arm for the cutting device we believe the use of auxetic materials for example in the form of handed sheared auxetics (HSA) (Lipton et al.
2018) could be a viable option.
Four HSA cylinders can be combined to create a 4-DOF actuator that can move up and down, side to side, and front to back, and twist left and right.
Another viable option would be to implement a detachable gripper similar to the suction cup end-effector by (Lehnert et al. 2017).
Instead of designing the soft endeffector as an active-close system, the fingers could be active-open, meaning they are fully closed until actuated by a tendon.
This would allow to actively open the fingers, move the gripper into a pre-grasp position, release the tendon, and let the fingers passively grasp the pepper.
Once the gripper has a stable grasp we could detach it from the arm, leaving it connected only via the tendon.
We can then freely move the cutter into a different position to cut the peduncle.
While less complicated with regard to mechanical design and control, the sequential nature of this approach could be much slower than using a fixed gripper with an actively actuated cutting device.
Additionally, oscillating blades such as used by (Arad et al. 2020) could be an advantage because suitable cuts can be made from a much wider range of approach angles.
Conclusion
Building a fully automated robotic system for green pepper harvesting is challenging especially without any modification to plants or environment.
In this case study we demonstrated that soft end-effectors are a suitable low-cost choice for picking green peppers without causing damage to the fruit and plant.
Future works include further improvements to the end-effector design, the implementation of a detection and grasp planning pipeline, and the development of an actively actuated cutting mechanism or detachable gripper.
Acknowledgments
The authors would like to thank Ajay Nair, Taylor Mauch, and the Department of Horticulture at Iowa State University for providing access to their green pepper field and for sharing their knowledge about planting and harvesting peppers.
We would also like to thank Dominik Guri and Benjamin Tarr for their help with assembly and operation of the robot.
This material is based upon work supported by the AI
Research Institutes program supported by NSF and USDANIFA under AI Institute: for Resilient Agriculture, Award
No. 2021-67021-35329.
Appendix End-Effector CAD files and Closeups