# Expanding omopgenerics

## Introduction

omopgenerics defines an `ecosystem` of methods and classes particularly the class that can be expanded. Currently there are two packages that define cdm sources:

- omopgenerics defines the source that defines the implementation of a ‘local’ (in memory) cdm source.
- CDMConnector defines the source that defines a general implementation for DBI connections.

In this vignette we explain how to expand the omopgenerics ecosystem defining more sources.

## The source object

First we need to define a function to create our source object: the source object must be an object (usually a list) that contains several attributes that will be used in the methods to fulfill their purpose. Finally we have to assign a class to our source and validate it with `omopgenerics::newCdmSource()`. The function has an argument to assign a `sourceType` that must be a character vector that identifies the name of the source. This is what will be retrieved by the `omopgenerics::sourceType()` function and it will be useful to identify how the source of the cdm_reference has been created.

Example how the creation of a new source would look like:

    myCustomSource <- function(argument1, argument2, ...) {
      # pre calculation and validation of arguments
      ...

      # create the source object
      obj <- list(x = x, y = y, ...) # this way you would access the attributes like: obj$x
      # or
      obj <- structure(.Data = list(), x = x, y = y, ...) # this you would access the attributes like: attr(obj, "x")

      # assign class
      class(obj) <- "my_custom_source"

      # validation
      omopgenerics::newCdmSource(src = obj, sourceType = "my_custom_type")
    }

If the first function that we create is `myCustomSource()` the validation with `omopgenerics::newCdmSource()` will fail as inside the methods are checked to be defined and work properly.

## Methods

You will need to write 4 to 7 methods for your new `<my_custom_source>`:

- `insertTable` required To insert local data into your source.
- `compute` required To compute a ‘query’ into a table in your source.
- `listSourceTables` required To list the data present into your source.
- `dropSourceTable` required To drop a table from your source.
- `readSourceTable` recommended To read a table from your source.
- `insertCdmTo` recommended To insert a cdm into your source.
- `summary` recommended To summarise and report the properties of your source.

### `insertTable`

Purpose: To insert a local table into your source object.

Arguments:

- `cdm` The cdm argument will be your source object created with `myCustomSource()` function.
- `name` The name to identify that table in your source.
- `table` A local `<tibble>` to insert in your source.
- `overwrite` (by default TRUE), whether to overwrite if the table exists in the database.
- `temporary` (by default FALSE), whether the table must be temporary.

Output: The output of a insertTable must be a `cdm_table` so your function must at the end validate it with `omopgenerics::newCdmTable().`

Sketch of how the function should look like:

    #' @export
    #' @importFrom omopgenerics insertTable
    insertTable.my_custom_source <- function(cdm, name, table, overwrite, temporary) {
      # code to insert the table into your source
      x <- "...." # it must be a reference to your table

      # validate output
      omopgenerics::newCdmTable(table = x, src = cdm, name = name)
    }

### `listSourceTables`

Purpose: To list tables that are present in the source.

Arguments:

- `cdm` The cdm argument will be your source object created with `myCustomSource()` function.

Output: A character vector with the names of tables present in source, empty identifiers `""` will be eliminated by `omopgenerics`.

Sketch of how the function should look like:

    #' @export
    #' @importFrom omopgenerics listSourceTables
    listSourceTables.my_custom_source <- function(cdm) {
      # code to list the tables present in source (cdm)
      x <- "...."

      return(x)
    }

### `readSourceTable`

Purpose: To read tables that are present in the source.

Arguments:

- `cdm` The cdm argument will be your source object created with `myCustomSource()` function.
- `name` Name to identify the table in your source.

Output: The output of a readSourceTable must be a `cdm_table` so your function must at the end validate it with `omopgenerics::newCdmTable().`

Sketch of how the function should look like:

    #' @export
    #' @importFrom omopgenerics readSourceTable
    readSourceTable.my_custom_source <- function(cdm, name) {
      # code to read the table 'name' from source.
      x <- "...."

      # validate as cdm_table
      omopgenerics::newCdmTable(table = x, src = cdm, name = name)
    }

### `dropSourceTable`

Purpose: To drop a table from your source.

Arguments:

- `cdm` The cdm argument will be your source object created with `myCustomSource()` function.
- `name` Name identifier for the table that you want to drop.

Output: The output is ignored, would recommend to return the source.

Sketch of how the function should look like:

    #' @export
    #' @importFrom omopgenerics dropSourceTable
    dropSourceTable.my_custom_source <- function(cdm, name) {
      # code to drop the table `name` present in source (cdm)

      return(invisible(cdm))
    }

### `insertCdmTo`

Purpose: To insert a cdm to your source.

Arguments:

- `cdm` A cdm reference from a different source. Recommend to collect each table before inserting.
- `to` The ‘to’ argument will be your source object created with `myCustomSource()` function.

Output: The output should be the cdm reference object inserted in your `to` source.

Sketch of how the function should look like:

    #' @export
    #' @importFrom omopgenerics dropSourceTable
    insertCdmTo.my_custom_source <- function(cdm, to) {
      # example of how it can look like:
      tables <- names(cdm) |>
        rlang::set_names() |>
        purrr::map(\(x) omopgenerics::insertTable(cdm = to, name = x, table = dplyr::as_tibble(cdm[[x]])))

      omopgenerics::newCdmReference(
        tables = tables,
        cdmName = omopgenerics::cdmName(x = cdm),
        cdmVersion = omopgenerics::cdmVersion(x = cdm)
      )
    }

The content of the function can vary depending of your source.

### `summary`

Purpose: To summarise the metadata of your source object.

Arguments:

- `object` The ‘object’ argument will be your source object created with `myCustomSource()` function.
- `...` For consistency.

Output: A named list of metadata of the source, each element must be a string of length 1.

Sketch of how the function should look like:

    #' @export
    summary.my_custom_source <- function(object, ...) {
      # extract metadata
      metadata1 <- "..."
      metadata2 <- "..."
      metadata3 <- "..."

      list(metadata1 = metadata1, metadata2 = metadata2, metadata3 = metadata3)
    }

### `compute`

This function works slightly different to the rest the input it will be a query instead of the source object.

Purpose: To compute a table into a permanent placeholder in your source.

Arguments:

- `x` Query to compute.
- `name` The name to identify the resultant table in your source.
- `temporary` (by default FALSE), whether the table must be temporary.
- `overwrite` (by default TRUE), whether to overwrite if the table exists in the database.
- `...` For consistency.

Output: The output of a compute must be a reference to your table in your source data, it will be converted later to a cdm_table (but you do not have to worry about that).

Sketch of how the function should look like:

    #' @export
    #' @importFrom dplyr compute
    compute.my_custom_source <- function(x, name, overwrite, temporary, ...) {
      # code to compute the table into your source
      x <- "...." # it must be a reference to your table
      return(x)
    }

## The cdm reference object

Finally every `<cdm_source>` class object would also need a function to create a to do that you just have to read all the tables that you want to include in your cdm object. tables must be a list of with the same source.

    cdmFromMyCustomSource <- function(argument1, argument2, ...) {
      # read and prepare the cdm tables
      ...

      # return the cdm object
      omopgenerics::newCdmReference(
        tables = tables, # list of cdm and achilles standard tables
        cdmName = "...", # usually provided as input, but also you might want to search in the cdm_source
        cdmVersion = "..." # either "5.3" or "5.4"
      )
    }

If you want to add to your object do it after the initial cdm creation like:

    # read from source
    cdm <- readSourceTable(cdm = cdm, name = "my_cohort")

    # or insert from local
    cdm <- insertTable(cdm = cdm, name = "my_cohort", table = localCohort)
    cdm$my_cohort <- cdm$my_cohort |>
      newCohortTable(
        cohortSetRef = cohort_set, # table with the settings of the cohort_table
        cohortAttritionRef = cohort_attrition, # table with the attrition of the cohort_table
        cohortCodelistRef = cohort_codelist # table with the codelists of the cohort_table
      )

This step can be included in your cdm object creation if you wish:

    cdmFromMyCustomSource <- function(argument1, argument2, ..., cohortTables) {
      # read and prepare the cdm tables
      ...

      # return the cdm object
      cdm <- omopgenerics::newCdmReference(
        tables = tables, # list of cdm and achilles standard tables
        cdmName = "...", # usually provided as input, but also you might want to search in the cdm_source
        cdmVersion = "..." # either "5.3" or "5.4"
      )

      # read cohort tables
      readSourceTable(cdm = cdm, name = cohortTables)
    }