Beam Blockage and Wind Farms solved with Radar AI
When people talk about usage of artificial intelligence on meteorological data, the first thing that comes to mind is nowcasting, a short term precipitation forecasting. However, there are many areas where it can be applied. In general, machine learning is best applied where it outperforms statistical or analytical methods, either taking considerably less time to compute while still providing high quality results, or simply outperforming those methods altogether.
One common problem with information acquisition using a radar are non-ideal surroundings. Especially a presence of hills, forests or tall buildings in its vicinity. This means that the radar has an imperfect view around it and some azimuths may be partially or fully blocked, resulting in incorrect or no data being scanned.
One way to deal with this issue after recognizing it is by replacing scanned data with filled in values. However, there currently doesn't exist a simple method for this data interpolation problem that would achieve both accurate and realistic looking results. Hence, it is a good candidate for usage of machine learning, where we can leverage the power of neural networks to tackle this task.
Problem definition
The newest use of machine learning at Meteopress, which we call beam blockage filling, is a clever way for us to deal with this issue. First thing needed in this process is an identification of blocked areas. This information is an expert knowledge that is usually reported by engineers who install the radar and observe its functionality. However, when a radar has been already installed for some time, there is another method for this identification. This can be done by calculating aggregations of scans over a long period of time. The image below shows aggregations of various elevation scans of Mt Gambier radar operated by Australia's Bureau of Meteorology office. It can be clearly seen that a large portion of north-east area is obstructed and the data is missing.
To have those areas filled in would be very valuable, as it would allow for finer data being obtained, more accurate predictions being done and better warnings being sent out.
Solution description
The idea is to utilize powerful capabilities of neural networks to fill in these areas. We have opted to using a model specifically designed this task of image inpainting, which we have adjusted to our needs. This is a model that contains modified convolution layers that are able to distinguish valid data from the missing ones, called Gated Convolutions. The problem with traditional convolutions is they treat all pixels in an image equally, regardless of if they are valid or not. The learned gating mechanism is what allows the network itself to learn the optimal way to extract information about data validity. Additionally, the model training is enforced to not only create accurate data corrections by using direct pixel-wise image-to-image comparison, but also make them look realistic by defining the problem within a generative adversarial network framework. What this means is there are two neural networks being trained concurrently. A generator who's goal is to correct the data by filling in missing (invalid) areas. Then there is a discriminator, a neural network who's goal is to distinguish generated data from real ones. The discriminator's output is then used to punish the generator for making corrections that the discriminator deemed as "fake".
First thing we needed to come up with is what kind of data we will use for training the neural network. Since this problem could be partially looked upon as regression, the network would need to compare its outputs to something we expect it to do, called targets. This means we could not simply take the data from radars that have those obstructions and use them for training. What we had opted for is using data from radars that have no obstructions and artificially create these obstructions on this data using a semi-random approach. This would ensure that the network would not be trained only for a specific blockage, but would be robust enough to be able to generalize for an obstruction of any direction and reasonable size.
The input to the model are five last scans of the lowest four elevations of the radar and a mask of same size containing information of which data is valid or invalid (obstructed). The model would then output those elevations with corrected areas that had been marked as invalid. The valid areas are not changed, since changing valid data is not desirable.
Results showcase
It turns out that there are several use cases for the model. First and foremost, it is able to correct and fill in missing data from blockages. It was able to both learn the structure of meteorological data, as well as stay consistent for various intensities of echoes. The showcase below is from already mentioned Mt Gambier and it shows the lowest elevation, since that's where most echoes in these situations were visible, and thus is the most challenging to fill in accurately.
This is a big step forward in filling incomplete radar scans. We had been trying out various frameworks, but most of them suffered from the issue of not keeping the consistency of the data over time stamps, meaning it was clearly visible that these areas had been filled in. However, as can be seen above, this time it appears that that is no longer the case, since one must really try and notice any discrepancies of within the fillings.
Moreover, it turns out that a network of this architecture can be applied to another case directly, something called a wind farm echo correction. On the lower elevations, the blades of wind turbines can sometimes be visible as stationary echoes, as stated in the information about Adelaide (Buckland Park) radar. When the model is applied for this situation, it can be seen that it is able to correct those areas, whether they are isolated, or they interfere with actual meteorological data. There are several wind farms that may be visible for this particular radar, so the specific one for each situation had been encircled to highlight it. It can be see that the model had either suppressed the echo from wind turbines in situations where there is meteorological data overlapping them or removed them completely when no echoes are visible, other than the wind farms themselves.
Conclusion and future work
We have been able to successfully develop another software that improves the quality of data obtained via meteorological radars. The network makes corrections in matter of seconds when run on a single GPU, which is very important as well.
The next step is to have our forecasters evaluate this data and point out any discrepancies or suggest improvements that could make the corrections even more accurate or beneficial. When all of this is done, the software will be available to use together with our radars by default.