How Important to build a Data Quality framework in your Cloud Data Lake

Data Quality Framework is mainly to create a quality data warehouse by integrating multiple systems across your organization and make it easily accessible to the users through various dashboards.

The overall DQ process will have the following stages.

DQ Roles

The DQ Process involves the following roles that take part in Rule Authoring, Rule Approval and Rule Execution and reporting

Data Owners

People or team in the organization that are ultimately accountable for maintaining the quality of a defined set of data

Approves the rules defined or requested by Data analysts and customer requests.

Data Stewards

Management of the data elements — both the data itself and the metadata

help operations team members ensure they follow rules, guidelines and standards when entering technical metadata into Rule Master JSON.

Data Consumers

Data users who use the data regularly.

They are responsible for defining what standards data must be held to in order for it to be useful. They are also the first line of defense in identifying and reporting data issues to the team.

Data Quality Ops Team

Maintains the technical metadata into Rule Master JSON

Data Quality Development Team

Implements the Data Quality rules in rule engine