Data Sources

Data Sources represent the original and intermediate systems that feed work performed in Optra. In this implementation, the original source system is Microsoft Dynamics 365 (D365). Client data is first ingested from D365 into the client's Databricks bronze layer, where the raw operational data is landed and retained in its earliest analytics-ready state. This bronze layer becomes the controlled handoff point between the source application and the governed data workflows built in Optra.

ODA does not connect directly to D365. Instead, ODA accesses the data through Delta Share, using the shared Databricks bronze data as the upstream source for Data Studio projects. This is an important architectural boundary: D365 remains the business system of record, Databricks bronze serves as the client's ingestion and persistence layer, and Delta Share provides the governed sharing mechanism that exposes the data to ODA in a controlled way.

Within Optra, Data Sources therefore act as the bridge between the Delta Shared datasets and the authoring experience in Data Studio. When users work with sources in projects, they are effectively working with data that originated in D365, was landed into Databricks bronze, and is now being made available to ODA through Delta Share for inspection, transformation, semantic modeling, and downstream job execution.

From the backend perspective, Data Sources are modeled as persisted source records and source nodes that participate in project lineage and ingestion execution. The backend stores source-specific metadata such as path, database, schema, warehouse, Delta Sharing details, and schema field information. When a source is opened or refreshed, the application can fetch schema details and sample data, normalize those fields for display, and preserve them so later pipeline steps use the same source definition consistently.

Data Sources also drive connector-based ingestion behavior. During pipeline execution, the backend reads the source definition, resolves the connected system and secret configuration, and routes the ingestion through the correct connector implementation. In this flow, Delta Sharing is the key access mechanism that allows ODA to consume the client-managed Databricks bronze data without bypassing the governed sharing boundary.

In addition, Data Sources are indexed for discovery and search. The backend builds searchable source metadata and schema-field metadata so users and services can locate relevant data across account, workspace, project, and node scope. This makes Data Sources not only an ingestion primitive, but also a discoverable asset in the broader Optra platform, supporting search, lineage context, and downstream automation.