Updated the Data Vault implementation & automation training for 12-14 June in...
On the 12th-14th of June I will be delivering the newly styled and updated Data Vault implementation and automation training together with Doerffler & Partner. I am really looking forward to...
View ArticleClik here to view.
When a full history of changes is too much: implementing abstraction for...
When changes are just too many When you construct a Point-In-Time (PIT) table or Dimension from your Data Vault model, do you sometimes find yourself in the situation where there are too many change...
View ArticleClik here to view.
Using a Natural Business Key – the end of hash keys?
Do we still need Hash Keys? Now there is a controversial topic! I have been thinking about the need for hash keys for almost a year now, ever since I went to the Data Vault Day in Germany (Hamburg) end...
View ArticleUpdated sample and metadata models for Data Vault generation and virtualisation
After a bit of a pause in working on the weblog and technology (caused by an extended period of high pressure in the day job) I am once again working on some changes in the various concepts I’m writing...
View ArticleBiml Express 2017 tests, comments and work-arounds
The new version of Biml Express, the free script-based ETL generation plug-in for Visual Studio provided by Varigence, has been out for a few months. Mid-July 2017 to be precise. However only recently...
View ArticleClik here to view.
Embrace your Persistent Staging Area for Eventual Consistency
If you like your PSA so much… A colleague of mine asked me this: ‘if you like the Persistent Staging Area (PSA) concept so much, why not embrace it all the way?’. By this, he meant loading upstream...
View ArticleClik here to view.
Beyond ETL Generation & DWH Virtualisation – what’s next?
Bridging the gap At the recent Data Modelling Zone (DMZ) in Germany I presented an overview of the ideas around Data Warehouse Virtualisation and the thought processes leading up to this. In this post...
View ArticleClik here to view.
Running SSIS packages continuously without scheduling
No more Batch ETL A few weeks ago I wrote a post about the concept of having continuous execution of ETL individual processes to achieve ‘eventual consistency‘. In that post I made the case to step...
View ArticleSome Q&A on Data Warehouse Virtualisation
I receive a fair bit of questions on the Data Warehouse Virtualisation ideas and wanted to respond and discuss this via this post. I don’t have all the answer but can share my views and expectations....
View ArticleClik here to view.
Is Data Vault becoming obsolete?
What value do we get from having an intermediate hyper-normalised layer? Let me start by stating that a Data Warehouse is a necessary evil at the best of times. In the ideal world, there would be no...
View ArticleAdopting GitHub for documentation, and resulting blog changes
After having used Git(Hub) to work and collaborate on code for a long time, I have recently spent some time to merge and move various documentation artefacts to GitHub as well. This covers the Data...
View ArticleRegistration now working!
I’ve finally properly (I think) configured the website to allow registration and the adding of comments in a user-friendly way, without having the burden of endless spambots. Registration, the creation...
View ArticleUsing (and moving to) raw data types for hash keys
Making hash keys smaller A few months ago I posted an article explaining the merits of the ‘natural business key‘, which can make sense in certain situations. And, from a more generic perspective, why...
View ArticleNew improvement round for the Data Warehouse Automation ecosystem
Today, a new set of changes -all part of the ecosystem for Data Warehouse Automation– have been formally published as new releases on Github. As always a large amount of work has been done in the...
View ArticleNew releases for open source Data Warehouse automation
Today I spend some time finalising and testing new releases for the ecosystem for Data Warehouse Automation, specifically the Taxonomy for ETL Automation Metadata (TEAM) and the Virtual Data Warehouse...
View ArticleClik here to view.
Why a delete flag makes sense (and when it doesn’t)
What are ‘logical’ deletes? Recently I was engaged in a few (separate) conversations about the use of deleted flags, and I thought it may be beneficial to post a summary here. A deleted flag, or...
View ArticleRoelant Vos to join the Varigence team!
After many years I have finished up at Allianz to join the Varigence team, so that I can work on the BimlFlex solution for data solution automation. Automating data solution has always been my...
View ArticleClik here to view.
An effective lightweight automation approach for Azure Data Factory
Last week I started at working for Varigence to work with the team on the BimlFlex solution for Data Warehouse Automation, so time to revisit some techniques in the Microsoft space. While doing so, I...
View ArticleThe BimlFlex Community
One of the things to ‘solve’ working for a software vendor is how to balance delivering meaningful content for the community for collaboration purposes with commercial software development and sales....
View ArticleClik here to view.
How to agree to disagree (on data warehouse automation)
This is a verbatim of my presentation at Knowledge Gap 2021, about ways to collaborate on data warehouse automation. In this presentation, I present the ideas and application of a schema that can be...
View Article