Data solution metadata schema – v2.0
The updated (v2.0) version of the data automation metadata schema has been finalised! This version is\ the culmination of many discussions over a long period of time, and hopefully is a step in the...
View ArticleClik here to view.
Joining tables in the Persistent Staging Area
In this blog post, I would like to share a pattern for joining tables in the staging layer of the data solution – specifically in the Persistent Staging Area (PSA). It is something that comes up...
View ArticleClik here to view.
Visualising the merger of time-variant and bitemporal data
In the previous post covering bitemporal concepts, we used a Data Vault example in which two satellite tables were combined to create a dimension table. When bringing together these two tables, some...
View ArticleClik here to view.
A not-so-gentle follow-up on bitemporal data challenges
Not long ago, we met Bob, in an introduction on bitemporal data delivery and how a well-designed data solution can provide this feature out of the box. When delivering data from the integration layer...
View ArticleClik here to view.
A gentle introduction to bitemporal data challenges
Your data is wrong! No, but we’re just looking at it from different points in time. A flexible data solution, such as a data warehouse, has the capability to look at data from different perspectives....
View ArticleClik here to view.
Deterministic dimension keys in a virtual Data Vault
Summary By extending the Data Vault patterns with a row number, called a ‘data delivery key’, it is possible to define deterministic keys for downstream use. This approach leverages the order of data...
View ArticleClik here to view.
The Engine – a status update and roadmap
The concept of the ‘Engine’ for data solution automation, as described in detail here, covers various tools, frameworks and concepts. These ‘engine components’ require to work together in cohesion, as...
View ArticleTrying out Data Vault code generation just got even easier!
To facilitate ongoing research in tweaking Data Vault patterns for various use-cases, I recently updated the open source data warehouse automation environments TEAM (source-to-target mapping...
View ArticleClik here to view.
How different modelling approaches impact your Data Vault
Data Vault methodology, and by extent all Ensemble Logical Models, are designed to be flexible. Approaches such as these accept that the model, as an interpretation of reality, is always changing....
View ArticleGlobal Data Summit presentation – using BimlFlex with a Business Model
Here is a recording of my presentation on using BimlFlex for code generation based off a business model, for the Global Data Summit 2021. With a brief intro by Hans Hultgren.
View ArticleKey metrics for monitoring data logistics
A data logistics control framework is an essential component for any data solution. There are many flavours out there, both custom-developed or provided as part of off-the-shelf software. For...
View ArticleClik here to view.
Generating Data Warehouse Automation schema code in Azure DevOps
Previous posts have gone into detail on using templating engines for code generation, and standardising on design metadata (inputs) for this and other use-cases using the schema for Data Warehouse...
View ArticleClik here to view.
Generating data logistics using Biml with the schema for Data Warehouse...
This post details how you can connect your own Biml automation framework to the schema for Data Warehouse Automation. A focus on being technology agnostic, and repository-less The schema for Data...
View ArticleClik here to view.
Easy Automation for Azure Mapping Data Flows – part 2
In the first post of this series, we have configured Azure Data Factory to accept an list of data logistics definitions so that a single template can run any number of processes. This approach for...
View ArticleClik here to view.
Easy Automation for Azure Mapping Data Flows – part 1
In the realm of Microsoft Azure, the Mapping Data Flows feature of Azure Data Factory (ADF) is the visual data logistics alternative. It can be seen as the cloud solution that predominantly supports...
View ArticleClik here to view.
Investigating code generation for Delta Lake using Azure Data Factory
Introduction – back in code generation mode Things have been busy after joining Varigence a few months ago. A major focus has been to develop new code generation features, to enable the BimlFlex data...
View ArticleClik here to view.
How to agree to disagree (on data warehouse automation)
This is a verbatim of my presentation at Knowledge Gap 2021, about ways to collaborate on data warehouse automation. In this presentation, I present the ideas and application of a schema that can be...
View ArticleThe BimlFlex Community
One of the things to ‘solve’ working for a software vendor is how to balance delivering meaningful content for the community for collaboration purposes with commercial software development and sales....
View ArticleClik here to view.
An effective lightweight automation approach for Azure Data Factory
Last week I started at working for Varigence to work with the team on the BimlFlex solution for Data Warehouse Automation, so time to revisit some techniques in the Microsoft space. While doing so, I...
View ArticleRoelant Vos to join the Varigence team!
After many years I have finished up at Allianz to join the Varigence team, so that I can work on the BimlFlex solution for data solution automation. Automating data solution has always been my...
View Article
![Various Artists – Old School / New School [iTunes Plus M4A]](http://i.imgur.com/GEFz815.jpeg)
![[Prime] Osprey Daylite Travel Pack 35 - Nightshift Blue $135 (RRP $199)...](http://files.ozbargain.com.au/n/87/976987l.jpg?h=1a642bba)
