Tom Denton introduces A2O Search, a project I’ve been working on throughout the year:
Today, we’re expanding this collaboration with the launch of A2O Search. This new sound search engine, built with Google AI, allows researchers and scientists to search through millions of hours of Australian wildlife audio collected by the Australian Acoustics Observatory (A2O).
Traditionally, researchers analysed wildlife audio by manually listening through hours of audio clips to identify an animal sound. This painstaking and laborious process posed barriers for research and has hindered land management decisions. Even with the introduction of machine learning tools for classification, collecting training data is an enormous undertaking and limits the number and kinds of questions that scientists can answer.
I’ve worked on the front-end, which allows researchers to upload an audio recording to search for similar recordings in A2O’s database. The recording is visualised in a spectrogram and researchers can clip the recording to a five-second junk to narrow down the audio input. The results can be viewed and filtered by location, date or time of day.
The project received coverage in major Australian news outlets.
“What we have built here is a search tool to liberate the data collected in the field. Instead of trying to manually sift through what amounts to hundreds of years of data that we could not live long enough to go through, AI does it for us,”
Researchers hope the audio search engine will yield important insights into where some animals go after bushfires and other natural disasters, and to better understand the impact of climate change and the spread of invasive species.
IMPACT Unofficial writing about APT (Algorithm Publication Tool), a project I have been working on since the beginning of the year:
APT is designed to support the unique needs of scientific writing by creating a high quality user-friendly authoring tool that generates PDF and HTML versions of final documents. Further development has added more user-friendly capabilities. An ATBD author can seamlessly add complex equations using LaTeX; track ATBD versions; and automatically number and label equations, tables, and figures. Document citations are provided for all public ATBDs.
Mapbox GL JS’ latest release (v2.9.0) adds a new globe projection, which enables presenting web maps in a way that allows viewers to interact with geographic data much like you would interact with a physical globe.
This new projection displays the map as a 3d globe and can be enabled by either passing projection: globe to the map constructor or by calling map.setProjection(‘globe’). All layers are supported by globe except for Custom Layers and Sky.
The globe projection is a continuation from custom projections, which were introduced a couple of months ago and, for the first time, allowed creating interactive web maps using projections other than Web Mercator. While custom projections were ideal to present data restricted to countries or continents and without the hefty distortions that come with Web Mercator, the globe projection allows to do the same but on a global scale; ideal to present global data sets like visualisations of climate change.
Felt, a new web-based map editor, launched in public beta last week.
Felt isn’t just another web GIS; it’s a tool for collaborative map-making. You can drop pins (even using emojis as markers), plot routes and highlight areas on the map and can annotate all this with text, notes and images. But there aren’t any features typical for professional GIS software, such as editing attribute tables or capturing complex geometries and valid topologies. However, Felt supports importing data from various formats (KML, KMZ, GPX, and GeoJSON) and exporting maps to GeoJSON.
This is a tool for anyone to create maps, whether they have prior knowledge in GIS or not. It’s designed for citizen engagement and participatory mapping; it’s for communities, not professional surveyors. Quite similar to the work around participatory mapping that groups like UCL’s ExCiteS and Mapping for Change do.
I like the simplicity of Felt. It focuses on a well-defined use case and is well executed. Much thought went into Felt’s design; the routing tool is a great example. Wherever you click, it snaps to the closest road and automatically calculates the route between two points, so you don’t have to add nodes to follow bends or turns at every intersection. By holding the Shift key while drawing, you can also draw segments that don’t align with the road network.
The team behind Felt found a gap in the current product landscape and is addressing the need nicely. I’m curious where they will take the $15M Series A funding.
Placemark have released a neat map-data conversation tool that transforms data between pretty much any geo-data file format. Upload files or paste text, convert and then download the converted data in no time.
In the course of implementing lots and lots of file formats in Placemark, we’ve ended up with some great, reusable tools. I figured it’d be pretty useful to just let anyone use those things, on a convenient drag & drop (or click, or paste) page. I hope it’s useful. Happy Friday!
This list will go out of date, but right now - you can convert:
Tom MacWright explores whether newer geo-data formats, like FlatGeobuf, Zarr, GeoParquet, Arrow, or COGs, are useful for applications making frequent updates to the data.
The post dives deep into some of the characteristics of these data formats, including compression, random access, and random writes, and concludes that they are optimised for reading data and that the benefits for writes are limited:
I like these new formats and I’ll support them, but do they benefit a usecase like Placemark ? If you’re on the web, and have data that you expect to update pretty often, are there wins to be had with new geospatial formats? I’m not sure.
Flatgeobuf is a relatively new binary format to encode geographic vector data. Unlike vector tiles, Flatgeobuf is a lossless encoding format, which is streamable and enables random feature access.
Horace Williams wrote a comprehensive overview of what goes into a Flatgeobuf file and how it is structured internally.
A useful little tool from my friends at GIS•OPS: Select a point on a map and get the coordinates, nicely rounded to six digits, in lon/lat or lat/lon. I use latlon2clipboard all the time; it’s a tool I didn’t know I needed.
The additions break with a long-lasting custom in interactive web mapping. Web maps are typically rendered using the Mercator projection, primarily for practical reasons: A square map is easier to divide into tiles. And Mercator preserves angles which is essential for navigation, the primary use case for early web maps.
[…] Mercator heavily distorts sizes on a global scale, inflating shapes the farther away they are from the equator. This is why most people living in the Internet era think that Greenland is as big as Africa, even though it’s actually 14 (!) times smaller.
But non-conformal projections, in large scales, distort shapes towards the edges of the map. Mapbox’ implementation solves this problem by seamlessly transitioning from the selected projection to Mercator in higher zoom levels, minimising distortions at every zoom level.1 You get the best of both worlds.
The new map projections open up new possibilities for interactive thematic mapping, especially for maps of continents or the whole world.
I created an Observable notebook to demonstrate different projections, their configuration, and how a map transitions between projections while zooming. ↩
(carbon)plan are introducing a series of JavaScript libraries for visualising Zarr datasets on web maps, complete with React components for interaction and exploration of Zarr data.
Zarr is a new data format to store and query multi-dimensional data, primarily used by scientists to represent grids of numerical data, like results from climate modelling. The new libraries are Segway to making environmental data explorable on the Web and more accessible to people outside the scientific community.
Paul Osman’s socio-technical view of software explains why every system ever built is in a messy state that is difficult to navigate. It happens earlier to some projects and later to others — but it does happen eventually.
The systems we are building and operating are constantly being modified by different people, with different contexts, at different times, who may or may not speak to each other directly.
The longer a system lives the harder it gets to understand why a software is built and behaves in a certain way.
If you want to understand why your systems perform the way they do, it’s necessary to know how expertise is created and distributed amongst the people in your organization. The single best way to do this is to invest in incident analysis.
Incidents are only one reason software changes. Others include technological advancements, change in user behaviour, erratic product decisions, even personal preference of developers. These drivers will never be captured by incident write-ups alone. Constant documentation is required, outlining the context, reasoning and consequences of a decision. Ideally we can trace when a decision changes previous decision. In other words, use Architecture Decision Records to document changing software systems and to build a written history of decisions that you refer to long after people involved in those decisions have left.
Back in the old days, many blogs had a blogroll listing links to other blogs the author follows. Blogrolls were the way to discover new blogs and rediscover old ones. With the arrival of Tumblr, Medium, and Substack, which mostly replaced publishing on self-hosted and self-built websites, blogrolls disappeared almost completely.
But there’s hope. Matt Webb’s blogroll, for once, is an absolute treasure trove. It lists more than 200 blogs, most of them with RSS feeds, including many old gems I had forgotten plus many I had not yet heard of before.
Brandon Liu uses classic video games Myst and Doom to explain differences between discrete-zoom pixel maps and continuous-zoom vector maps and, based on that, what that even means: A fast web map.
Sound advice that the simpler, less fancy solution is often the more appropriate choice:
In the same way pre-rendered images were good enough to make Myst a best-selling game, non-realtime rendering of maps is good enough for some, maybe even the majority, of web map use cases today. If you’re building a whizbang mapping demo with 3D flythroughs to show your company execs, a real-time-rendered system - the Doom approach - is the best choice; if you’re building a site for helping people find the nearest COVID vaccine, the simplest solution is the best solution, and that usually means stepped-zoom maps.
A new release of Leaflet.Deflate is now available. Leaflet.Deflate 2.1 introduces a new option, greedyCollapse, to specify when features will collapse.
With greedyCollapse set to true, a feature collapses when either its width or height falls below the specified threshold. This is the default, and it matches the behaviour from previous versions. Set greedyCollapse to false, and features will collapse only of both width and height fall below the specified threshold. This behaviour comes in handy when you have many “long” features on your map.
I started my career at Development Seed, and was part of the team that spun out and became Mapbox. And back in the day, everything was GitHub issues.
The company internal blog was an issue-only repository. Blog posts were issues, you’d comment on them with comments on issues.
Sales deals were tracked with issue threads.
Recruiting was in issues - an issue per candidate.
All internal project planning was tracked in issues.
Development Seed still uses GitHub Issues almost exclusively to manage the day to day operations.
It works because GitHub has everything you need to plan work and see what’s moving from different angles, even for complex projects with several people working in the team. Project leads use Kanban-style project boards to see where tickets are in the pipeline. Milestones are ideal for planning and tracking sprints. For personal to-do lists, a list of tickets assigned to you, you create custom views of your work using filters, unique URLs to bookmark included.
I never understood the appeal of heavy-handed project-management software. If your code already lives in GitHub, you might as well save some money and use GitHub’s project management tooling.
Brett Cannon making a point, grounded in philosophy, about the relationship between open-source maintainers and users of their software
When you treat a maintainer as a means to getting something from their software you are not morally treating them appropriately as an end (in other words, you’re simply using them), and thus not treating them morally as a human being. But when you treat a maintainer as a fellow human being who may be able to do you a favour of their own volition, then you end up in an appropriate relationship where you are not trying to use the maintainer for something specific.
And to conclude:
There should be no expectations toward the next commit, next release, etc. when you realize open source maintainers really don’t owe you anything. If you view open source code from that perspective then you will view it as a gift when it exists at all. As such, hopefully you feel less frustrated when open source doesn’t go the way you want since it was all a gift to begin with. And that then will lead you to treat maintainers as an end in and of themselves and thus as a fellow human being.
There is work that looks like overhead for a personal project or seems to hold you back at the beginning of a professional project but will pay off later in the product lifecycle. Simon Willison calls them PAGNIs — probably are gonna need it.
When should you over-ride YAGNI? When the cost of adding something later is so dramatically expensive compared with the cost of adding it early on that it’s worth taking the risk. On [sic] when you know from experience that an initial investment will pay off many times over.
Test automation, continuous integration and automated deployment are non-negotiable PAGNIs. API pagination, time-stamping records, and logging API traffic are others I didn’t think of initially but are just as important.
Logging request payloads proved to be especially valuable in previous projects I worked on. Tracking down bugs that only affect certain users is almost impossible if you do not know what data clients send to your backend. Plus, we were able to recover hundreds of transactions in the aftermath of an incident.
Paul Ramsey, the creator of PostGIS, looks back on the early days of the spatial database extension for Postgres:
When Dave had a working prototype, we hooked it up to our little applet and the thing sang. It was wonderfully quick, even when we loaded up quite large tables, zooming around the spatial data and drawing our maps. This is something we’d only seen on fancy XWindows displays on UNIX workstations and now were were doing it in an applet on ordinary PC. It was quite amazing.
Around 2004, after exclusively doing GIS only in ArcInfo, I worked on an online atlas for a local environment agency. We used PostGIS, MapServer and the Rosa Java applet in the front-end, all duct-taped together by some pretty awful PHP code. Nevertheless, there were solutions now that allowed us to put interactive maps on websites. Being able to store and retrieve data from a performing open-source database was one of the foundations that made modern-day Web GIS possible.
I‘ve hardly worked with any other spatial database since, other than occasionally toying around with CouchDB or SpatialLite.
I want to do work that matters, and what matters more than making geographic data accessible to humans? Working at Development Seed is a continuation of the journey I started as a Geography student 20 years ago and left about two years ago to try something different.
I‘m happy to be wrangling latitudes and longitudes and making maps again.
I published release 1.1.0 of netlify-plugin-html-validate. The only notable change is the upgrade of html-validate from 4.6 to 4.10.1, which makes available several changes, most notably two new features:
In addition to .htmlvalidate.json, you can configure html-validate using .htmlvalidate.js, which enables comments and better interoperability between sharable and regular config files.
Note: This release is not yet available in Netlify’s plugin directory, while Netlify’s engineers are working on a solution to support breaking changes in plugins. You can add the latest release to your build using file-based plugin configuration.
‘Embrace the grind’ is one of the best pieces of advice you can give to early-career software developers.
Successful developers are rarely the ones who know every architecture or design pattern, every last detail of a programming language, or every JavaScript framework currently en vogue. Great developers work through complex problems that require several iterations to solve. They go spelunking into complex code they didn’t write to understand the root of a nasty bug or spend days breaking down a significant feature into small deliverables so that each stage adds value. It requires grit and determination. Earlier in my career, this looked like magic, but it got easier over the years; it’s a skill you develop.
Winds circulate around the globe, forming patterns of gigantic proportions. These patterns become part of human culture and are reflected in our architecture. They are hidden designs, mapping the complexion of the earth, which we can uncover. By orienting on the direction of general winds, airports recreate wind patterns, forming a representation of a global wind map with steel and stone, thus making the invisible visible.
The predominant westerlies in central Europe, even local winds like the Mistral along the Rhone valley in southern France or the Sirocco in northern Africa and southern Europe, are all reflected in the layout of airport runways.
You can take stewardship too far. I probably have. There is a diminishing return on what you can get out of any system, regardless of how much you put into it. But I don’t care.
[…]
And I’m able to emerge and become human, or part human, and enter into that line-by-line world, where there is very little judgement, just you and the mechanics of the systems, systems that become increasingly beautiful the more time you spend with them. For me, this stewardship is therapy.
Some people maintain a garden, not to win a prize at a gardening show or to grow and sell the most delicious apples. They do this for themselves to look at and enjoy the slightly too sour apple every now and then. I write and maintain this website, knowing that only a tiny portion of internet citizens will ever land on these pages.
I can spend hours working on this website’s tiniest details, making things simpler, sometimes a little more complex. I would write a script to make it easier to create new posts, make the site load a little faster, or tweak the copy on the about page. Making progress on a project for the sake of progress alone — it can be therapeutic.
In 2017, Chris Cox, Facebook’s longtime chief product officer, formed a new task force to understand whether maximizing user engagement on Facebook was contributing to political polarization. It found that there was indeed a correlation, and that reducing polarization would mean taking a hit on engagement. In a mid-2018 document reviewed by the Journal, the task force proposed several potential fixes, such as tweaking the recommendation algorithms to suggest a more diverse range of groups for people to join. But it acknowledged that some of the ideas were “antigrowth.” Most of the proposals didn’t move forward, and the task force disbanded.
The metrics a business chooses to measure its success drive decisions and define its product. You can use all the marketing words in the world to frame your work in a different light, but eventually, it‘s always about what you measure.
If you‘re optimising user engagement to make people watch episode after episode of a TV series, then you’re selling TV shows. If your product automates brokering loans to small businesses, you’re in finance. If you’re building a product that optimises engagement, so users see and click more ads, then you’re in advertising. You‘re not building communities or bring the world closer together. And you surely aren’t making the world a better place.
I published a new release of Leaflet.Deflate. Leaflet.Deflate is a plugin for the web-mapping library Leaflet. It improves the readability of feature-heavy maps by converting features to markers on smaller zoom levels.
Version 2.0 includes a breaking change and a couple of more minor improvements:
Following a previous release that introduced a new way to inject layers to control how deflated features are displayed, I removed options markerCluster and markerClusterOptions from the API. This change will affect you if you are using Leaflet.Markercluster to group markers. To upgrade to the new API, initialise a MarkerClusterGroup and inject it via the markerLayer option. Follow the example code to see how it’s done.
The unpacked size of the package is now 42.8 kB — down from 10.9 MB. I removed the examples and other files which are not necessary to install and run the library. All examples can still be accessed via the repository.
According to NPM’s download statistics, a significant number of developers still run older versions, including the now four-and-a-half year old 0.3. I have updated the README to make it easier to find and access documentation for older releases. You really should upgrade to a later version, though, if you’re still using 0.3.
When location data is linked with other data about people and the world we live in, we can gain important insights and create new services that greatly improve how we live, work and travel. With these new data applications and opportunities, there are emerging privacy and ethical considerations. So that we can continue to benefit from widespread use of location data, it is important that data is used in a way that mitigates concerns and retains public confidence.
Sounds like the government is helping businesses to understand how much location data they can collect and cross-link with other personal information without pushing it too far. Shouldn’t the government instead work on legislation to protect citizen’s personal information from the claws of international corporations?
The project’s oversight group includes, among others, representatives from Telefónica and Mastercard. Both companies don’t need location data—other than the data they already have—to be successful. To provide their service, Telefónica need to know how many people connect to a cell tower at any given time and how much bandwidth these people require. Telefónica doesn’t need to know where I drink my beer and what transport I use to get there. Likewise, the only information Mastercard need to worry about is whether I pay back my credit card debt but not if I get my suit from Savile Row or my local Marks & Spencer.
My former colleagues at University College London, Artemis Skarlatidou and Muki Haklay, have edited and published a new book:
Geographic Citizen Science Design takes an anthropological and Human-Computer Interaction (HCI) stance to provide the theoretical and methodological foundations to support the design, development and evaluation of citizen science projects and their user-friendly applications. Through a careful selection of case studies in the urban and non-urban contexts of the Global North and South, the chapters provide insights into the design and interaction barriers, as well as on the lessons learned from the engagement of a diverse set of participants; for example, literate and non-literate people with a range of technical skills, and with different cultural backgrounds.
Looking at the field through the lenses of specific case studies, the book captures the current state of the art in research and development of geographic citizen science and provides critical insight to inform technological innovation and future research in this area.
I published a new release of netlify-plugin-html-validate. 1.0 looks like a big release; only it is not.
There is only one significant change included in this release: An upgrade of html-validate from 2.23 to 4.6 to include new validation rule sets. The upgrade introduces a potential breaking change. html-validate’s config will no longer automatically extend html-validate:recommended. As a result, you may need to update custom configurations of html-validate. html-validate’s release notes contain full information and upgrade guides.
Thanks to Chris Buckley for contributing to this release.
This release is not yet available on Netlify’s plugin directory for installation via the Netlify UI. Netlify’s plugin system cannot handle breaking changes to plugins gracefully at this point. Their engineers are actively working on a solution. You can add the latest release to your build using file-based plugin configuration.
