Build a Data Mesh with Dataplex: Challenge Lab ERROR

[AlexHeringer]

Hi experts.

I'm trying to finish the Build a Data Mesh with Dataplex: Challenge Lab.

I'm consistently receiving the following error: 
Rule named 'NOT NULL' not found for the user_id column in the data quality task. Please re-create the task including this rule as mentioned in the instructions.

Could anyone help me? I already tried to change the rule name to NOT_NULL, NOT NULL,NOT_NULL_1 & NOT_NULL_2 and it always persists with the same error.  

I've checked the zone, the project id... everything is ok. In fact, it produces a valid data quality result, but i don't know why it don't find the rule to user_id, even if it exists and the results are published on big query. 

Follow below the yaml I'm using. 

metadata_registry_defaults:
  dataplex:
    projects: qwiklabs-gcp-03-2106b87ca1ea
    locations: us-west1
    lake: Sales Lake
    zone: curated-customer-zone
row_filters:
  NONE:
    filter_sql_expr: |-
      True

rule_dimensions:
  - consistency
  - correctness
  - duplication
  - completeness
  - conformance
  - integrity
  - timeliness
  - accuracy
rules:
  NOT_NULL:
    rule_type: NOT_NULL
    dimension: completeness

rule_bindings:
  USER_ID:
    entity_uri: bigquery://projects/qwiklabs-gcp-03-2106b87ca1ea/datasets/customer_orders/tables/ordered_items
    column_id: user_id
    row_filter_id: NONE
    rule_ids:
      - NOT_NULL
  ORDER_ID:
    entity_uri: bigquery://projects/qwiklabs-gcp-03-2106b87ca1ea/datasets/customer_orders/tables/ordered_items
    column_id: order_id
    row_filter_id: NONE
    rule_ids:
      - NOT_NULL

[tausahemad · Accepted solution]

Hello @AlexHeringer ,

Can you try below code in yaml file.
=======================================================
metadata_registry_defaults:
dataplex:
projects: $DEVSHELL_PROJECT_ID
locations: $REGION
lakes: sales-lake
zones: curated-customer-zone
row_filters:
NONE:
filter_sql_expr: |-
True
rule_dimensions:
- completeness
rules:
NOT_NULL:
rule_type: NOT_NULL
dimension: completeness
rule_bindings:
VALID_CUSTOMER:
entity_uri: bigquery://projects/$DEVSHELL_PROJECT_ID/datasets/customer_orders/tables/ordered_items
column_id: user_id
row_filter_id: NONE
rule_ids:
- NOT_NULL
VALID_ORDER:
entity_uri: bigquery://projects/$DEVSHELL_PROJECT_ID/datasets/customer_orders/tables/ordered_items
column_id: order_id
row_filter_id: NONE
rule_ids:
- NOT_NULL
==============================================

[AlexHeringer]

thanks for the reply. worked

[kmoorthy_5]

Hi!

I trying to finish the Build a Data Mesh with Dataplex: Challenge Lab:

Tried multiple times but no luck so far.

Rule named 'NOT NULL' not found for the user_id column in the data quality task. Please re-create the task including this rule as mentioned in the instructions.

here is my yaml:

metadata_registry_defaults:
  dataplex:
    projects: qwiklabs-gcp-04-22303233277e
    locations: us-west1
    lakes: salts-lake
    zones: curated-customer-zone
row_filters:
  NONE:
    filter_sql_expr: |-
      True
rule_dimensions:
  - completeness
rules:
  NOT_NULL:
    rule_type: NOT_NULL
    dimension: completeness
rule_bindings:
  VALID_CUSTOMER:
    entity_uri: bigquery://projects/qwiklabs-gcp-04-22303233277e/datasets/customer_orders/tables/ordered_items
    column_id: user_id
    row_filter_id: NONE
    rule_ids:
      - NOT_NULL
  VALID_ORDER:
    entity_uri: bigquery://projects/qwiklabs-gcp-04-22303233277e/datasets/customer_orders/tables/ordered_items
    column_id: order_id
    row_filter_id: NONE
    rule_ids:
      - NOT_NULL