​(00:00) Patty Skinkis

This is the Hi-Res Vineyard Nutrition Podcast series, devoted to helping the grape and wine industry understand more about how to monitor and manage vineyard health through grapevine nutrition research. I am your host, Dr. Patty Skinkis, Professor and Viticulture Extension Specialist at Oregon State University.

(00:22) Patty Skinkis
One of the main objectives of the High-Res Vineyard Nutrition Project is to come up with a sensor that can assist with nutrient monitoring in the vineyard. Ideally, the sensor should be easy to use, durable, and provide information that a grower can use to improve vineyard health and product quality, whether it’s wine, juice grapes, table grapes or even raisin grapes. In earlier episodes, I spoke with Dr. Jan van Aardt and Dr. Alireza Pourreza, both of who are co-PI’s on the High-Res Vineyard Nutrition Project, and who is working with hyperspectral imaging of vineyard trials. These trials are set up by viticulture researchers in projects throughout Washington, Oregon, California, New York, and Virginia. These research plots have adjusted nutrient levels by the researchers and are scanned by the engineering team using their sensors to determine differences in nutrients based on vine canopies. Joining us today is Dr. Qin Zhang, the project co-director leading the engineering team from WSU. His team at Washington State University is focusing on sensor development and testing vineyards in the Pacific Northwest. Dr. Qin Zhang is Professor of Agricultural Automation where he serves as the director of WSU’s Center for Precision and Automated Agricultural systems located in Pressor, Washington. His area of expertise is mechatronics and automation, and much of his research focuses on agricultural automation such as sensing for data analytics for many crops including tree fruits and grapes.

(01:55) Patty Skinkis
Thank you for joining us today, Qin.

(01:57) Qin Zhang
Yeah, thanks for inviting me to this event, and I’m so happy to share some of our research with you guys.

(02:06) Patty Skinkis
So we’ve had the pleasure of speaking with other members of your research team as I mentioned Jan van Aardt and Dr. Alireza Pourreza prior to us sitting down and talking, and they talked a lot about the hyperspectral cameras as the best technology to use for nutrient sensing. Are you using similar sensors in your studies in eastern Washington?

(02:28) Qin Zhang
Yes, we do use this. This is actually a very mature sensing technology for agricultural production, especially to detect the nutrient stress for all different crops, and you know the history of that: you can go back 40 years when precision farming technology was being developed, and that the core technology is using sensors or sensing technology to detect the crop nutrient condition. Then they create a plan how to manage the nutrition. I would say this technology has been used very successfully for agronomic crops. For example, corn, wheat, rice, etc. But when applied to the grapevines there could be some technical issues or challenges to be addressed.

(03:32) Patty Skinkis
So why hyperspectral cameras? You mentioned there’s a long history, they have been used for 40 years. What specifically makes them really valuable in nutrient sensing?

(03:44) Qin Zhang
OK, this is a very good question. Maybe I should talk a little bit about the fundamentals of sensing (the hyperspectral sensing technology). The hypothesis is that the sensor can detect crop and nutrient conditions. We assume the canopy color is a representation of some chlorophyll level of the crop and which is an indication of crop nutrient stress. We have a lot of different ways to measure those levels, but most of those measurement means are destructive, and we need to collect samples for the crop and that takes time to make an analysis. This hyper-spectrum sensor is a wonderful sensing technology that you do not need to destruct. So, we have non-destructive sensing technology. So, how it works is it measures the reflectance of some specific light from the crop canopy, and then, based on the reflectance estimates the crop nutrient condition. So, often we call it an indirect, approximate sensing technology. It has an advantage that is non-destructive, and it can be done remotely, so, often we assume that we have already heard a lot about satellite-based or drone-based remote sensing. So, this is a technology that allows the sensor to be carried by satellite, by drone, or by tractor, or it is hand-held. So, this is the only technology you can practically use in such a way today.

(06:00) Patty Skinkis

So, largely those visual cues that we can see with our eyes are also what these cameras are able to look at, and specifically identify an issue. Now, you mentioned with these other crops where it’s been used, how it’s being used in the application for those other crops?

(06:21) Qin Zhang
In my hand, I have a dealership survey to show the current condition. Not a current – let’s just say two years ago, because all the surveys, the kind of, we have a backlash. So, the latest data I have is from 2020. OK, for 2020, if we use the imaging sensing technology to detect the other crops, I mean, the agronomic crops--corn, rice, wheat--it’s about 70% of the product applied is a satellite-based or aerial-based imagery technology. I mean aerial is a use of an aircraft. So, it’s oftentimes a high elevation, low resolution, but it covers a larger area at once. Then, with the joint technology being UAV. We often times call it “UAV,” but people also call it “joint technology.” With the joint technology being intensively used in recent years, over 40% of farmers have already adopted this technology in their operation. I mean in the agronomic crops. Then, in recent years the tractor or sprayer-based sensors are also increasing but compared with satellite-based or drone-based, the adoption is still low. The reason for that is that this is useful for real-time applications. So, what does “a real-time application”? We put a sensor in front of the sprayer and the sensor detects the crop nutrient condition in real-time, and then it goes back to the controller of the nozzle to determine how much nutrient (fertilizer, and water) we need to apply. So, this is in an early stage, so it’s about 13% of the variable rate, or the farmers are using variable rate, the sensor-based variable control for that.

(08:40) Patty Skinkis
Oh, that’s great statistics on how it’s being used in and in other crops and hopefully, in the future, we’ll see that more in the perennial cropping systems such as grapes. So, some of the technology in the vineyards, and some other research that’s going on such as “The Intelligent Sprayer” is using LiDAR. Is this something that’s also being included in this sensing technology as a means to measure canopy size and not just wavelengths? Because we know that every time, we tell growers to look at nutrient sampling we want them to take a sample, and test it, but also make a comment on canopy size, vine vigor, etc. Is there a way that the sensor can also integrate some sort of measure of canopy size?

(09:21) Qin Zhang
Yes, actually, there is some research on “The Smart Sprayer” we are talking about, and just as you said, it uses LiDAR to detect the canopy size. But this is just for the size-based Smart Sprayer. So, if we have a very big or larger canopy. This is more or less not for nutrient application but for disease or insect control. So, if we have a large canopy, then we need to apply more [pesticide] and if we have a small canopy, we will apply less. So, those kinds of sensor technologies are for different information. For nutrients, we need to find the crop nutrient stress. For that, right now, we’re not able to do it. You can detect the size, but not the nutrient content. But on the other hand, if we do decide to make a sprayer application based on [canopy] size, then the hyperspectral sensor may not be able…I would not say unable to do this very easily or not the best, most effective way to do that. So, different sensors are for different applications, or for different uses.

(10:52) Patty Skinkis
Sure, they are different sensors. However, is there the possibility that they [sensors] could be combined?

(10:57) Qin Zhang
That’s right. Yeah, so often we say the smarter machine is based on the integration of different
sensors to get all the needed information to support automated field operations.

(11:13) Patty Skinkis
At this time the research that you’re doing is focusing just on the wavelengths, just on the nutrient-sensing part not the canopy size part, correct?

(11:24) Qin Zhang
That’s right. Actually, in CPAAs and in our center we have literature to cover both. But for this the particular project we just focus on the wavelengths for nutrition detection.

(11:36) Patty Skinkis
So it’s a good thing to know that the technology exists and as we work forward in the future, there’s probably a way that those can be paired on a final product or a sensing tool, whether it’s on your sprayer or on your tractor or on your applicator.

I’m going to go back to talking about the wavelengths and these visual cues. Since plants have only a few visual cues they can indicate problems: there’s chlorosis, so the yellowing of leaves, there’s necrosis, so the dried, brown tissues, and there’s red coloration and maybe some other variations. But it's really those are the main visual cues in terms of color. How does the sensing technology confirm a nutrient issue rather than a virus issue, for example? Because we know in vineyards, we can have both going on. We can have viruses and we can have nutrient issues and we can have water issues which can all compound with each other.

(12:33) Qin Zhang
This is very true, and because of the visual cues, as we all know, those color changes, or whatever the changes are, it may not just be caused by one factor. It’s a combination of all factors. But the use of this is, if you ask if the sensor can distinguish that? My answer is “yes or no”. I say “yes” based on several reasons- one, those symptoms may not be occurring at the same time and then we can distinguish the timing. This is one way to do that. The second way is with a spectrum that means it’s combined by many wavelengths. Ok, by our human eye, we can see the broad spectrum of all the visible light we can distinguish, and the sensor is better than the human eye because it can separate different wavelengths. So, for example, some nutrients could be more sensitive in some specific wavelengths instead of just broadband for the combination of the wavelengths. The same as some for the virus or diseases or even water stress. So, one goal for this research is that we use a hyperspectral sensor to collect the baseline data and hope to find out those sensitive wavelengths for nutrients. Then we select those wavelengths and develop the sensor most sensitive to nutrient stress rather than a virus, or other diseases, or even water. But if you say, “Can we totally distinguish or find a sensor only sensitive to nutrients based on today’s technology?” I have to say no, we cannot by today’s technology. But as technology advances, there are some other technologies in the early stage of research, for example, some wearable sensors, which is to detect not as a refraction of the light, but some other biological or physical parameters of crop growth, and then those sensors can be installed on a leaf of a crop. Then, with the combination of integration with those sensors, we could develop a sensing system which is only sensitive to the nutrients, rather than a specific nutrient. But this is still under research and development.

(15:40) Patty Skinkis
So I like that description of kind of the suite of products to use or sensors to use and I think that helps to remind us of all that these sensors are just sensors. They determine something but we still have the human knowledge behind them. Like you said, with whether it’s red blotch or leaf roll, they’re both red and phosphorus can also be red leaf. But knowing the timing when we normally see them show up will help us inform what the sensor sees. So, I think that’s a critical reminder for a lot of people that the sensors can’t do everything. They just do a very small part of that equation to help us do things better, but we can inform all of this with what we already know from the plant response. But also, that component of taking it to that next step to build models and inform it through artificial intelligence, I think, is going to be that next step.

(16:38) Qin Zhang
That’s absolutely true. That’s absolutely true.

(16:40) Patty Skinkis
So it’s a suite of products and it takes time to do all of that research and it’s exactly why we’re talking about this today to just share where we are at the research. I thought it would be good to have you tell us a little bit about the research projects that you’re doing specifically for the Hi-Res Vineyard Nutrition Project. For example, what are you evaluating specifically and what type of grapes are where, specifically, you’re doing some of the research?

(17:11) Qin Zhang
Ok, you know, earlier I said this particular sentence: this technology is very mature for agronomic crops and is being adopted widely over the past 20-30 years, but the adoption to​ grapevines is still questionable because different crops have different features. For example, for corn: if we see a corn canopy from the air, we can basically see the entire canopy. The young leaves and the older, fully developed leaves. So entire nutrient distribution could easily see from the air, from the top view. But for vine grapes, for example, if we see from the top view, we can only see those young leaves, very young leaves. And are those young leaves carrying accurate indications of the nutrient stress or not? We do not know. So, this is one of the reasons why I told you that for this particular project, we propose to assess and evaluate if the current mature sensing technology can be adopted directly. If it could, then life is easy. But more likely, we cannot because of the different nutrient distribution features for the grapevines from the field crops. This makes it challenging. So, how to handle that? OK. We can develop a model based on just young leaves to detect the true condition of the entire plant. Or, we need to use a side view? Or we need to use a combination with an angled view? OK. Those are things that we’re not sure. So, this research is basically for us to help to find those answers. This is why, for this particular team, we have two groups. One in the East [US] and one is in California [Western US]. They are, more or less, trying to find out if we can use the top view, just like for any other crops. Could we find an accurate estimation model to assess the qualifying nutrient condition? And then, the Washington team, we just basically try to address the more fundamental issue: how different is the top view from the side view? And is there any relationship? And if there is some consistent relationship existing, then those kinds of relationships can be used to develop the top view model, using human intelligence and so forth. Based on the field crop adoption experience taught us that the most economical way to use that kind of sensor is from the top view because we can deploy a satellite, aircraft, or drone. The use of the field, on-ground-based sensor, is not as effective or efficient as the top view because ground sensing is limited by all kinds of limitations, and the view window is very small. But before these issues can be addressed, it would be difficult to develop a trustworthy top-view model to integrate those sensor data. Just like you said earlier, sensor data is sensor data. If I had the urge to say, in our car, we have speed sensors, we have speed indicators, we have temperature indicators, it depends on how many gauges you have on your dashboard. And all those use sensors to pick up the information. But what information are those sensors picking up? It is an electronic signal. The sensor output is just an electronic signal-- its voltage change or current change. If we resolve to use a model to interpret a signal, a sensor signal, those signals are meaningless. We must develop a trustworthy model to interpret those signals and present that in a way we can understand for the particular application.

(22:12) Patty Skinkis
So what do you think is the biggest success that the engineering teams have had in this project so far? And, I know we just started really in 2020 and so it’s relatively new but what do you think is the biggest outcome or finding from your group?

(22:30) Qin Zhang
I want to say that it’s been two seasons even though this season is not quite finished yet and the team and all three teams collected a reasonable number of samples for baseline. And those baseline samples already told us some hypotheses or suspicions we had before we started. Let’s say, for example, the top leaf did make a big difference for us to integrate the real situation. And so, we do need to develop a trustworthy model before we can really use the drone or satellite data to interpret the situation. And ways to collect the data, in usual analysis, we do see that, and we do see some patterns exist. I would say the biggest accomplishment is that we proved our hypothesis is right. We are going to the right direction to make the sensors and the sensing technology applicable or adaptable for grapevine industry.

(23:56) Patty Skinkis
That’s great! So, what do you think, amongst all of those successes, what do you think was the biggest challenge from a research perspective?

(24:05) Qin Zhang

The biggest challenge would be the data--to collect sufficient data. Actually, before the sensing technology matured for field crop application, I was involved in developing similar sensors for corn when I was working at the University of Illinois 20-30 years ago. At that time, many universities and companies worked together to collect a huge database, and then develop the model to support those sensors. Right now, based on my knowledge, there are very few teams working on this [for grapevines]. That means we can collect very limited data, and just as with any science or any model-based science or data-based science, the accuracy, and the trustworthiness of the model very much depend on the availability or the completeness of the data. For example, for a sensor, per se, it is extremely important that raw data or the baseline data covers the entire spectrum of our interest level. For nitrogen, per se, if we say we’re interested in the 100% satisfied, or non-stress, with 100% stress, then we would like the data points to cover the entire span of the spectrum. Based on our current study we can only select a few data points. Even for each point, we may collect hundreds or even thousands of images, but that’s only one point. And we may need thousands of points to fill the gap. And the more data you have, or the least gap you have, the more accurate the model will be. That’s one thing.

And the second thing, the second challenge is, the grapevines are biological objects, and the challenge is, just like for a human being, it is very difficult to find two human beings exactly the same. And this is also true, is it very difficult to find two vines that have the exact same signature, even with the same nitrogen stress, let’s say, or nutrient level. So this causes a lot of variations because the same variety planted at a different location of the same vineyard could be presented differently. So, this creates another sensitive point of view which is a noisy signal. How to “de-noise”? This is also a big challenge. Before making a model trustworthy, de-noising is very, very important. And this de-noise depends upon a huge database needed to do that. This was the second challenge, and the third was different varieties. They may be presented differently, and different growing seasons may be presented differently. What we’re finding out for the field crops is that what you measure today in the same field with the same stress level or nutrient level, and what you measure tomorrow could be different. And the change is continuous. And then how to handle this? This will be the third challenge. Because we can develop a model, but is this today’s model or tomorrow’s model? This could be neither. Maybe we need a multi-dimensional model. And at the end of the day, could it be one dimension of this model? But again, the challenge is--how to collect sufficient data for each day?. Or do we have sufficient data for each dimension? So, this will be multi-dimensional: timing, variety, and even location. The location could be in the same vineyard or the location at a different place, for example, Oregon, Washington, or California, these could all be different. So, a real challenge is that we do need a huge database to support the model development. This model will be either multi-dimensional or a group of models, each model suitable for each different application. Say for different varieties, we have a model. For different regions, we may have a model. So, this is still in the research. And then with AI technology, I would say, to some of those products with the help of AI development, I think all of those challenges could be addressed with the condition if we do have sufficient data to support.

(29:50) Patty Skinkis
So it sounds like we have our work cut out for us. We need many sites, many years, and good data. I think that’s so important for the listeners to know that even with technology, it sounds like the answer is to streamline everything. But to come up with that answer takes a lot of work and a lot of that ground truthing. So, I appreciate that you mention that.

So now I’m going to switch gears a little bit and I’m going to ask you how you decided to become an agricultural engineer. You know there are a lot of different fields of engineering, how did you decide on agriculture?

(30:26) Qin Zhang

This is a very good question, and actually, quite a few people ask me this question and why I picked ag engineering as my career. Actually, before I went to college I worked on a farm for a few years, and I did see the importance of technology to agriculture. So, when I went to college I decided to pick mechanization because at that time a major challenge where I grew up was we didn’t have enough power to farm efficiently. At that time, we didn’t even have a tractor. We just used the hoes and did all the farming by hand where I was from. So, I picked Ag engineering as my subject to study in hope that it would bring sufficient farming power to help the farmers to do that. And then, after getting into the profession, the more I got into it, the more I felt like I made a good decision. First of all, many people think agricultural engineering or agriculture may not be that fancy, which is totally not true. Agricultural engineering is as fancy and as challenging as any engineering discipline. In fact, ag engineering adopts almost all other forms of engineering disciplines and tries to use their findings or their technologies to solve the issues in farming. Because farming itself covers so broad of an area for the challenges in which we can face any kind of technological help. That’s one thing, and the second thing is, I would say agricultural engineering or agriculture is an essential industry for human beings. Unless, one day, maybe, somebody creates or modifies a human being where we don’t need to eat anymore. Otherwise, agriculture is essential. Let’s say we don’t have computer scientists, people can still survive, but not as effective. But let’s say without agriculture, without crop growing, people would immediately disappear because we will starve to death. So, this is what I see: ag engineering is fun, it’s essential, and it feels a lot of challenges. One time, actually, quite a few times I was invited to give talks to computer scientists at international conferences and at first, they told us, “you know, we can solve everything for you guys.” I thought, “No, no, no, no, this is not true.” To solve agricultural issues, you have to have an understanding of how agriculture is conducted. If you have never seen how grapes are grown, managed, and harvested, and processed; and if you only know how to drink, you will never be able to solve the problems of how to grow grapes. After that chat, they understood the challenge and they saw why ag engineering is important and fun.

(34:17) Patty Skinkis
Oh, that’s great. I agree with you, I think it’s a very challenging field to work in, and I’m glad that people are interested. So, when you’re seeking out students or students of agricultural engineering, do you see many students coming into this area of agricultural engineering, and/or what would you say to students who are thinking about getting into engineering, or maybe don’t even know they have the aptitude? What would you say are the strong aptitudes for agricultural engineering?

(34:49) Qin Zhang
I would say, first of all, it’s like I said earlier, agricultural engineering is important and essential, it feels challenging and enjoyable. That’s the most important: it is enjoyable. But unfortunately, I do see there’s a decrease in young people interested in ag engineering. When I was in college, 40 years ago, we had a lot of ag engineering departments all over the world. Today, the number of ag engineering departments is dramatically reduced. Both Oregon State University and Washington State University do not have undergraduate programs for ag engineering anymore. It used to be that both universities had very strong ag engineering undergraduate programs. I do not know why, but this is maybe very likely because of a lack of students coming in that are interested in this type of career, which I say is not quite true. So today if you look at the ag engineering profession and imagine them coming in from a background of mechanical engineering, chemical engineering, computer engineering, electrical engineering, and because after they got involved in ag engineering research, they say it’s really fun t​​o work, it’s really enjoyable, and so forth. But again, this does not address the future need for agriculture. So, I still wish more and more young people would like, will see this opportunity, and will be willing to work in agriculture. Currently, we can solve this problem because, if we use myself as an example, many of our researchers, and faculty members are not U.S. born. Actually, all of this engineering team is not U.S. born. This is a challenge. I would like to wish more and more of our kids would be interested, especially our kids from a farming background, to see that this is a big opportunity. It’s very enjoyable to work, it’s very important and very rewarding.

(37:35) Patty Skinkis​​
Well, that’s great to hear, I agree. I think we need more people who have that background of growing up on a farm, knowing how that application comes in, because they don’t need to learn agriculture, they can step into engineering or other areas. So, thank you. I appreciate your comments about the research and about career and agricultural engineering. Thank you for joining us today and sharing your information. To learn more about Qin’s work and his team, you can check out the Center for Precision and Automatic Agricultural Systems’ (CPAAS) on their website, which is https://cpaas.wsu.edu/. As always, you can learn more about the Hi-Res Vineyard Nutrition Project at https://highresvineyardnutrition.com/ and you can follow us on Twitter. LinkedIn, and Instagram.