The fruit and vegetable sector is a key economic activity across Europe as a whole, standing out for its high productivity, diversification and specialisation, and for its role as a driving force for sustainable development in rural areas, in terms of employment, business opportunities and entrepreneurship. However, market globalisation, increased competition, declining profitability, and labour shortages and a lack of generational renewal pose significant challenges that affect the sector’s viability and sustainability in the medium and long term.

With the aim of addressing these technical, financial and labour-related challenges in the sector, the European FRUCTHOR-IA project is currently being developed. Its main objective is to improve the sector’s competitiveness and sustainability through the development of autonomous robotics and Artificial Intelligence (AI) solutions designed for crop monitoring and human-robot collaboration.

CATEC is taking part in this innovative initiative as one of its technology partners and project beneficiaries, contributing its extensive experience and expertise in autonomous robotics, computer vision, AI and the validation of technological solutions in real-world environments. More specifically, we are involved in the development of advanced technologies focused on the design, development and optimisation of autonomous robotic solutions for crop monitoring using autonomous systems that integrate advanced sensors, AI and data analysis.

Work is focusing on computer vision algorithms and the autonomous movement algorithm for a quadruped robot, both in terms of its trajectory through the crops and its vertical movement for the comprehensive monitoring of trees or plantations.

Furthermore, CATEC is conducting a pilot trial on farms in the La Vega del Río Guadalquivir region (in Lebrija, Seville), validating the use of the autonomous quadruped robot equipped with RGB sensors to select the optimal harvest point, in collaboration with companies from the local agricultural sector. This pilot focuses on fruit and vegetable crops such as tomatoes and peppers, with the aim of advancing solutions for productive monitoring and supporting decision-making in the field.

Progress in specific technologies

At CATEC, we are currently working on a number of technological developments. Firstly, a software architecture based on ROS 2 has been defined to integrate the various functional blocks of the robotic system, and a human-robot interaction system has also been developed using gesture detection, utilising the Intel RealSense camera on the Unitree Go2 robot, body and hand pose detection models, and a set of commands such as start, stop, cancel, move forward, turn and return.

Furthermore, progress has been made on a mission monitoring HMI, which allows real-time visualisation of the farm map, the planned and travelled route, the robot’s position via GPS-RTK, the mission status and geolocated events associated with potential diseases, pests or water stress.

Another key area of work is the creation of a realistic fruit and vegetable growing environment in Blender and Isaac Sim, based on tomato and pepper crops. This environment enables the simulation of the robot’s navigation and the validation of algorithms prior to field deployment, as well as serving as a basis for generating synthetic datasets. Using these environments, RGB images, segmentation masks, metadata, bounding boxes and depth data are being generated to train and validate detection and segmentation models.

In the field of perception and AI, segmentation tests have been carried out on leaves, fruits and stems, including trials using real greenhouse images, and work is also underway on photorealistic textures of healthy leaves and those affected by various relevant diseases. All of this will enable progress towards the early detection of anomalies and the generation of disease or yield maps.

Finally, CATEC is working on the integration of 2D and 3D vision using ZED X cameras, with object detection via YOLOv8 and projection onto point clouds to estimate the dimensions and volume of fruits or other elements of interest. With a view to the pilot trial, the aim is to integrate the perception, navigation, control, HMI and communication modules into the Go2 robot, collect real-world data from tomato and pepper crops, and validate the system during the growing season.

As a result, these technologies will be able to help generate heat maps of disease and production, estimates of harvests and yields by area, early warnings of infectious outbreaks and, in the future, recommendations for action relating to irrigation, treatment or targeted prescriptions.

Project partners and duration

In addition to CATEC, the project consortium includes CAR-CSIC (Centre for Automation and Robotics, Spanish National Research Council), as the lead organisation, the Castile and León Institute of Technology (ITCL), the Dordogne Chamber of Agriculture, the Dordogne Departmental Council, Agerpix Technologies SL, the École Supérieure des Technologies Industrielles Avancées (ESTIA), the Association of Municipalities of Cova da Beira (AMCB), the Polytechnic Institute of Bragança – CeDRI, and the Polytechnic Institute of Viseu (ESAV).

FRUCTHOR-IA is a project with a budget of 1.7 million euros and is co-financed by the INTERREG VI-B South-West Europe (SUDOE) programme through the European Regional Development Fund (ERDF). It is scheduled to run for three years, from 2025 to 2028.

Further information on the project is available at: https://interreg-sudoe.eu/proyecto-interreg/fructhor-ia/