Sometimes, even when you’re close to your team, patterns or underlying issues emerge over time. Some of these are insights you may wish you had recognised earlier. The important thing is to talk to your team and work on a solution together. Sometimes, together you resolve one challenge and enhance something else at the same time Like many teams, we’ve been through many Pull Request (PR) process evolutions. We started with requests in Microsoft Teams chats - they often got lost. We tried automated notifications in Teams, but they were only possible in an odd place - so they got ignored, and lost. Then we created a chat for our team just for PRs, with a strict no chat policy, and an emoji system so engineers can see someone is looking at their PR, when there are comments, when it’s approved, and the engineer can even mark it as urgent if necessary. It worked really well, until something unforeseen happened. The Challenge I realised that one of my engineers was taking on the ma...
On Tuesday this week, I was back at AWS on Holborn Viaduct learning about Apache Iceberg with a colleague. Iceberg is a table format standard for managing large-scale datasets and their metadata. The AWS-managed approach is to store the data in S3, meaning you only pay for storage and bandwidth. Other tools, such as Apache Spark , can then be used on top to build a data warehouse - or in this case, a Lakehouse. AWS also supports self-hosting, but this requires you to configure and manage aspects such as maintenance yourself. One of the major advantages Iceberg has over other data warehouses is its support for True ACID transactions, and its table format standard means you don’t have to run a database engine. Anything which wants to access the data can go straight to S3. As a software engineer with a background in object orientation, this feels potentially brittle and a coupling nightmare, but I am sure it works in practice. The pre-workshop presentations made Iceberg feel...