frankie-tales

SM-Tools, Copernicus, FOSS and the reasons of inevitable choices and drifts

2026-02-25T15:00:00Z

Here at work, we develop a series of tools for geospatial processing with multiple goals, expected maintenance durations, different scopes, generalization needs, and motivations. Note that about 10 years ago, here in Europe, a completely new approach to upstream and downstream services for Earth Observation began: ESA changed its data licenses, and distribution and access modalities entered the Big Data era.

That even changed the data approach for academic institutions and triggered a major shift in the daily work of researchers, with access to a tremendous volume of weekly data available almost just in time and worldwide. Of course, such a change also impacted us, and we had to adapt our processing and storage capabilities to the new era.

One of my side projects in that regard is SM-Tools, which consists primarily of a collection of support tools for running our internally developed soil moisture algorithm using SAR satellite data (SMOSAR). One such tool (now named smt_copernicus) began (and evolved with multiple restarts-from-scratch) more than 10 years ago, when the Sentinel constellation started operations. Its purpose is to search for and download satellite products from Copernicus archives using multiple criteria, and to maintain an internal geospatial database of these products, along with all derived maps and ancillary data. This is only one component of a system that should be able to process large quantities of multi-source data on selected areas of interest, create downstream products, and calibrate and analyze results by comparing them with field data. The clear final goal is to achieve new findings in satellite data analysis, supported by extended processing worldwide, and to introduce new algorithms.

This is a long-term goal that unfortunately runs up against short- to mid-term difficulties of accessing archives that are not under our direct control. The sad reality is that in the last 4 years, the Copernicus archive access modality changed 3 times, and in the previous period, Copernicus also changed policies and modalities in progress (e.g., by introducing online and offline products, changing formats, etc.). Geospatial communities are small enough to encounter more practical difficulties than expected in such operational conditions, and this is now an almost weekly experience. We now have to chase other parties’ changes more often than we did in the past, rather than working on our own goals instead.

For instance, until 2023, the main package used for accessing the Copernicus archive was the Sentinelsat Python package (developed by an handfull of willing scholars starting in the summer of 2015). It became abandonware that year when people discovered that the protocol changes required rewriting most of the package from scratch, including all test code and mockings. That happened again at the end of last year. Incredibly enough, all access protocols have been an ongoing affair since 2023, even for non-secondary details, and required frequent adjustments to avoid unexpected breakage in download and processing pipelines. That for sure does not encourage community-supported FOSS solutions. Exactly in 2023, when I discovered our Sentinel-related tool had to be deeply changed because of the lack of the obsoleted Sentinelsat package, I decided that enough was enough and managed to migrate from Python to a fully self-supported Perl reimplementation: one of the things I always hated in the Python ecosystem is the excessive (for me and our purposes at least) speed in deprecation of consolidated packages and features, and the prospective of having to chase after unexpected changes in both Copernicus AND Python sides was out of question. My experiences with Perl have been much less annoying in this regard, with scripts running perfectly even after 20 years since they were written. Let’s consider this the old-school approach: if something is working, don't touch it without a more than valid reason, and even then, think twice before you touch.

In the meantime, around 2019, another independent effort started to support Copernicus access, along with a few other data providers. That’s about 4 years after the original Sentinelsat project. Timing in this case is essential to be considered. EODag is a single-company FOSS product that has been actively developed since then, but may have been considered stable 1-2 years later. Of course, it only provides the usual access layer, and using that package implied, at the time, replacing Sentinelsat with EODag as a base layer for searching and downloading only, while performing other tasks afterward with other self-made code. Before 2023, using Sentinelsat or EOdag was equivalent in order to perform the same task, with very little advantages.

Note that both tools were, in any case, pretty adequate but not enough for our goals, and also had a few defects (or a lack of flexibility, to say better) to manage in some creative way. That has been one of the reasons for not replacing Sentinelsat with EOdag in 2023. The other major one was that the idea of replacing a small package with another (as for Sentinelsat, there are just a couple of main contributors to the codebase, along with a good number of pending issues and PRs for such a kind of product) is probably not the safest one to avoid problems in the near future, when Copernicus will change things (again, see the next Earth Observation Processing Framework EOPF data format — Zarr). And of course, EODag is written in Python, and I already expressed my concerns about that.

Even if you like it or not, nowadays, the concrete alternative to adopting small FOSS projects to perform basic tasks is to use AI tooling to create a perfectly (or almost so) tailored implementation for the target task. While in 2023 I had to rewrite from scratch (in maybe a month of work with some fixes to Geo::GDAL::FFI package for a multi-threading issue) an implementation of a multi-threaded tool for accessing the Copernicus archive and maintaining an internal geospatial database of products consistently among multiple generations of the archive, I implemented the STAC protocol variant in a few hours instead, thanks to Claude Code-based patching and my reviewing and tests for the resulting codebase. As said in this post, currently, the cons of adopting a small third-party FOSS solution outweigh the pros, particularly regarding the resulting technical debt, compared with a well-conducted self-consistent AI-based development process.

I’m seeing in my own side projects exactly the mirror of what will probably be the reality of FOSS projects in the near future, in practice, as I mentioned in the previous post. Relatively few major/interesting projects will be adopted by others and attract contributions, while most codebases will become pure one-man shows, with AI tooling.

A significant part of geospatial processing involves data procurement and processing, i.e., refining and preparing data and images in order to collect, filter, and process large volumes of data for subsequent analysis. This is the most annoying and repetitive part of the process, and also often the most time-consuming. In my experience, working on those tasks is probably the most effective way to use LLMs through a spec-and-test-driven design. Whether you like it or not, it is the most immediate way to produce working code by iterating with a chain of thoughts and accurate reviewing of results, including a decent test coverage. As observed in Antirez's experience, the AI agent also had the nice ability to retain my Perl style (which is not secondary, given that Perl has many programming flavors and variants).

Maybe the final results will be an increase in quick-and-dirty codebases, but for many scholars, it will be a major simplification of their lives. In most domains of science, coding activities have been seen as an inevitable evil: they are a tool, not the primary goal, and even before the advent of LLMs, most scientific codebases were far from something to be proud of. The Copernicus attitude to FAFO will also encourage such approaches. Simply because scholars don't have time to waste chasing changes introduced by this or that data provider or company when contracts change.

AI sloping attitude? No, simple survival instinct.

A call to minimalistic programming

2025-09-10T17:00:00Z

Minimalism in development is a forgotten virtue of our time that should gain more attention. A straightforward summary of some minimalism principles is available here. Briefly, the principles of minimalism in Software Engineering can be summarized as follows, based on the manifesto for minimalism.

Fight for Pareto's law: look for the 20% of effort that will yield 80% of the results.
Prioritize: minimalism isn't about not doing things but about focusing first on the important.
The perfect is the enemy of the good: first do it, then do it right, then do it better.
Kill the baby: don't be afraid of starting all over again. Fail soon, learn fast.
Add value: continuously consider how you can support your team and enhance your position in that field or skill.
Basics, first: always follow top-down thinking, starting with the best practices of computer science.
Think differently: simple is more complicated than complex, which means you'll need to use your creativity.
Synthesis is the key to communication: we have to write code for humans, not machines.
Keep it plain: try to keep your designs with a few layers of indirection.
Clean kipple and redundancy: minimalism is all about removing distractions.

Most of those principles are coherent with each other and relate heavily to the well-known Unix KISS principle.

An extended and fascinating book about the practical application of such principles is Eric S. Raymond's "The Art of Unix Programming", which I strongly recommend reading. I can also recommend a now-classic volume on the same topic by John Ousterhout, "A Philosophy of Software Design". Both outline practical examples of how minimalism in design can be effectively embraced, with a focus on doing the right thing sooner rather than later.

The same principles could (or maybe should) be applied even to programming languages, but this is often a neglected aspect of such a minimalistic approach. Note that one of the most successful languages of all time is the C language, which indeed has a straightforward syntax and, as such, cannot be easy to use correctly (the principle is that what is simple is not necessarily easy, too). That's because the programmer needs to create her/his own abstractions and layers to build her/his vision of a software design. It seems that this is precisely the opposite of the C++ or Java approach, where the entire specification spans thousands of pages, and many high-level abstractions are integral parts of the language. The same can be applied to Python nowadays, which started as a simple language, more readable and clean than Perl, but now has a wide and articulated specification. Again, hundreds of pages are now needed to describe a once-simple language, where tons of new features and abstractions have been added to enrich its expressiveness. If one considers its standard libraries and modules, the actual situation appears even worse. Can such an approach be considered easier? I don't think so. Let me say: how can a program be considered simple if it relies on hundreds (or even thousands, including dependencies recursively) of external modules, as well as hundreds of syntactical constructs and glues? Some languages also manage multi-versioned dependencies, allowing a program to cross-depend on multiple editions of the same module (yes, JavaScript, I'm talking about you), with the concrete possibility of introducing obscure bugs as a result. At the opposite extreme, there is the consideration that we only know and deeply understand what we make.

Minimalism also means actively seeking a balance between these two opposing approaches, because reusing third-party modules and packages can be an immediate solution to deadline urgencies, but can also potentially introduce instability and dependencies on unmaintained software in the long run. Long dependency chains where changes happen independently of the main program focus and are introduced by third-party motivations and reasons - often with wrong timing for depending projects - can cause breakages at multiple levels.

Of course, to reach the right tradeoff, a few things need to be considered: every single programmer could not be smarter than a lot of libraries and modules out there, where multiple developers could have spent hours/weeks/months, or even years refining them. That's true, but it is also true that not all libraries or modules are written with the same level of quality and effort. For instance, we all know cases of elementary modules available for Node that could be easily avoided, and instead are imported for some form of laziness in development. Even, sometimes features that need to be used could be only a small portion of the whole library/module, which could be reimplemented with a very reasonable effort and time. This approach could be amplified in modern times when AI tools could significantly increase productivity in such cases. I would simplify these concepts with some additional mottos:

Limit your external dependencies: avoid depending on modules or libraries that are not strictly required to significantly reduce the total development time, are not rock stable for their interfaces and features, and do not have a clear and stabilized roadmap.
Reproducibility of the software stack is a must: these days, a SBOM has become recommended/mandatory, but it should not only consist in a documentation of external dependencies and their versions, but also the full process of building a runtime environment should be fully defined and consistent for the long term.
Do not follow the last oh!-so-cool technology: while that could be done for an amateur project to develop during spare time, it is not a good idea depending on a technology whose future is not clearly stated, with a well-established development team and proven sustainability in the long term. I consider a risk even depending on a single company project, and even more if it is considered a startup. Synthetically, this can be generically considered as minimalism in coding style.

Moreover, if you are going to use a well-established framework, such as Django, for developing your mid-to-long-term web project, it is probably better than using the latest Nodejs-based framework created six months ago that seems the latest 'big thing'. But that's probably only common sense. Instead, ask yourself if your project should be created from scratch using a simple jamstack system and some microservices for well-defined and minimal parts. In many cases, that is more than enough for too many CMS-based sites out there: indeed, I continuously ask myself why a lot of websites are still based on WordPress, when most of them could be easily converted into a handful of static pages and simple JavaScript snippets that they will use in any case. This can be declined in terms of minimalism in defining computing architectures, which can also allow scaling up applications more easily.

So minimalism principles can be considered at multiple levels: for programming languages, libraries, architectures, and design. However, they require skills, in-depth research, and a significant amount of time to dedicate to continuous refactoring and meditation about viable alternatives. And that's probably the key point: developers with deadlines and urgency imposed by PMs are too often tempted to follow the easiest and richest paths and provide a solution of any kind without too much meditation on the final balance among efforts, quality, efficiency, and durability of results.

Of course, about minimalism, an extraordinary citation is due for the whole suckless effort on the uncompromising minimalism side. And why not?.

Ok, ok, I'm joking. But you got the point.