The challenges of developing ML solutions for remote sensing imagery.
Developing a Software-as-a-Service (SAAS) Geospatial Machine Learning Operations (MLops) platform that can handle large remote sensing images is a complex and challenging task. The industry demands innovative solutions that can incorporate big data, machine learning, and advanced computing techniques to meet the needs of geospatial data experts. We would like to share with the community the difficulties and technical challenges we have faced in developing such a platform and the solutions we have implemented over the past four years.
1. How to process remote sensing imagery and why is it so challenging?
Remote sensing images are often captured in the GeoTIFF format, which is not supported by web browsers. This creates a challenge for the server to convert the images into a compatible format for users to view them on a web app like Deep Block. The conversion process must be efficient and fast, without resulting in a delay or timeout in the response time to the HTTP request. This requires background processing and notification to the user once the conversion is complete. To reduce the processing time, we have implemented patented algorithms and have optimized the platform's architecture to ensure quick response times, even for large remote-sensing images.
2. Once processed, how to display ultra-high-resolution images?
In some cases, satellite or aerial photos can exceed 10 GB, with a resolution exceeding 30K. This poses significant challenges when rendering the image on the browser. To handle this, we have implemented specialized algorithms that break down the image into smaller sections, which are rendered on the client-side using HTML's "canvas" component. We have also utilized various caching techniques to ensure quick access to previously rendered images and implemented specialized algorithms to ensure smooth and seamless rendering.
3. How to overcome canvas limitations?
As seen in the previous section, HTML's canvas components are used to render and manipulate images. However, the recommended maximum size of the image resolution that can be rendered on a single canvas is 5000x5000 pixels. To overcome this limitation, we have implemented specialized algorithms that split large images into smaller sections, which can be rendered on multiple canvases simultaneously. We have also developed innovative techniques that use pre-rendering to create an image mosaic on the server-side, which is then served to the client-side for viewing.
However, modern web browsers have memory usage limitations, with most web pages restricted to less than 50MB of memory. Past 128MB, the browser starts suffering greatly. This poses challenges when rendering ultra-high-resolution remote sensing images in our interface. Therefore, we only load the necessary sections of the image into memory, reducing the memory footprint of the application. Additionally, since we utilize cloud services to store and process these images, this frees up the client-side application's memory.
4. Then, why choose to develop a web-based Platform?
Despite web browser limitations, Deep Block’s SaaS Geospatial MLops platform must be web-based to allow users around the world to access the service. Otherwise, we would need to use CPP, or Python QT to make an end-user software, and except for CPP, other technologies are not popularly used. Web apps, on the contrary, offer infinite flexibility in layout and design and preset many benefits when designing and maintaining the platform
5. Why is parallel processing required to analyze super-resolution images?
As explained in a previous article, in order to analyze these super-resolution images, they must be divided into multiple sections. Division results in overhead since cutting this image creates multiple files, and processing these multiple images naturally requires more time than processing one image. We use parallel processing techniques to maintain high service speed. However, implementing such techniques is not easy and requires significant expertise. We have developed specialized algorithms to ensure that multiple instances of the same code can be executed simultaneously on multiple processors or nodes, reducing the overall processing time. In other words, we use distributed computing techniques to parallelize the processing of large geospatial images. This approach distributes the image processing tasks across multiple computing nodes, allowing us to process large images more quickly.
GeoTIFF images also contain geospatial metadata and objects in these images often appear small, requiring preprocessing and various algorithms to recognize them. To address this challenge, we have developed custom algorithms that preprocess the geospatial images to extract the relevant information. These algorithms help to identify specific objects within the image, such as buildings or vehicles, which can then be labeled and extracted for further analysis.
6. How to choose the right experts to support you?
The analysis of remote sensing images is a challenging task, given their large size and resolution, which make them a form of big data. Combining machine learning and technologies like parallel computing with big data exacerbates this challenge, and it is almost impossible for remote sensing practitioners to tackle it alone.
Furthermore, developing working software and implementing machine learning algorithms is not enough. The remote sensing industry generates innumerable images each day, and the ground is constantly changing. Therefore, the software must be user-friendly and able to analyze input images quickly to ensure a seamless user experience. Speed and user experience are crucial factors that researchers and individuals sometimes overlook, failing to acknowledge the importance of software engineering and computer software technology. If asked to design software architecture similar to Deep Block, they may not even know where to start.
The geospatial data industry is underdeveloped and heavily reliant on human labor for many analysis tasks, and classical surveying companies still dominate the market. Omnis Labs, with its expertise in computer science, can help address these issues by developing software that quickly processes images and is easy for users to operate. Although we may not have expertise in measurement, we can contribute to the remote sensing industry by delivering powerful software.
