Expanding horizons

Hi all!

I’m back with some news since my previous posts. After summer 2023 I had the privilege to work on a Synapse serverless / Spark based data lakehouse, and subsequently I took on a role as Cloud Data advisor for a large Cloud IT program at CIZ, a subsidiary of sorts for the Dutch ministry of Public health, wellbeing and Sports. So what, I hear you thinking:

The first assignment was my first as the hands-on lead, building a Spark based solution. As a SQL guy, this was big for me. And really cool: less code, more smarts, and more flexibility towards the future to incorporate third party libraries for next level data governance, quality management, master data management and smart analysis. Also, I got to work with family, which was awesome, my nephew did really wel. Junior in title, pretty senior by the way handled himself!

Secondly, as a cloud advisor, what even is that? A humble translation is Santa’s little helper, Santa being the program management team. Or, and this sounds way better, the executive branch of the program management team. It’s a quality assurance role, so I got to delve into aspects of IT delivery that weren’t running smoothly, whether information security, cloud (including Fabric) architecture, or software testing approaches. Since the program was executed at a governmental organization I got a good hard look into pretty strict IT security compliancy, IT project tendering and more, often at C-level, which I appreciate, as that’s where I can learn and develop myself.

All in all I’ve been busy, learned a lot of new stuff, most interestingly about myself and what drives me: building cool products and services, with a team, in an environment where I can learn and develop. Sounds familiar? What drives you guys these days?

Until next time,

Regards, Sjoerd


First business

After a long and quite relaxing summer holiday Dark Matters is going strong. I’m quite busy with my own business development, resulting in a nice funnel already. My first confirmed assignment is regarding creating and providing a training for several Microsoft BI consultants covering all of Fabric, the new Microsoft SaaS BI Data Analytics offering.

And Fabric truly is a different beast compared to the previous offerings! The most notable stuff in my view relates to:

  1. Fabric is truly lake centric. It’s kinda funny that Fabric labels the two engines as Lakehouse (Spark) and Warehouse (SQL), but in line with the more common industry definition, both are lakehouses, as they use OneLake and Delta as the persistence layer. So study up on data lake design and Delta!
  2. Fabric admin and licensing is a nice mash-up of the new Fabric and the old Power BI licensing schemes. Also, we now have ‘domains’, an organizational container on top of workspaces. This allows for a Data Mesh approach, where each domain is more or less self-sufficient, and delivers production grade information products to end-users or other domains. The need for a central hub in the mesh, e.g. for analytical master data, seems likely for larger deployments.
  3. The Spark compute engine is the first-class citizen now in Fabric, trumping SQL. SQL will be there for a long time, but still, time to study up on Spark execution (and debugging and performance tuning) and PySpark (and all the cool 3rd party libraries, and how to manage these dependencies).

There is much more to cover regarding Fabric, so stay tuned, and share your thoughts!

We’re live!

Welcome on my Dark Matters website. I’ll use this space to blog about my view on industry news and valuable analytics use cases: We all need some halo use case to convert people to our cause, don’t we? 😉

Keep watching this space…