Package com.isomorphic.tools

# Class DataImport

java.lang.Object
com.isomorphic.base.Base
com.isomorphic.tools.DataImport

All Implemented Interfaces:
  `com.isomorphic.base.IAutoConfigurable`

public class DataImport extends com.isomorphic.base.Base
`DataImport` contains methods that can be used to import data from a test data file to the List of Maps format commonly used for DataSource records (importToRows()), or directly into a DataSource (importToDataSource()).

By default the input file is expected to contain comma-delimited data like a `.csv` file, but JSON and XML are also supported.

By default, the input file is expected to be encoded as UTF-8. You can override this if necessary by using one of the versions of `importAndValidateDSRows()` that accepts a DSRequest, and passing "encoding" in the values on that request.

If you are using UTF-8 or UTF-16 encoding, by default the import process will automatically consume a Byte Order Marker if one is present in the stream. If for some reason you do not want this behavior, set the flag "import.consume.bom" to false in your `server.properrties` file.

Imported data may be transformed during import, for details search SmartClient Reference for `"dataSourceField.importStrategy"`.

See Also:
  - `ServletTools.loadWebRootFile(path)`

- ## Nested Class Summary

  Nested Classes
  Modifier and Type
  Class
  Description
  `static enum`
  `DataImport.ImportFormat`
  Import formats that can be used: `DataImport.ImportFormat.CSV` `DataImport.ImportFormat.JSON` `DataImport.ImportFormat.XML` `DataImport.ImportFormat.AUTO`
  `class`
  `DataImport.ParseBoolean`
  Implements the translate method of the DataTranslator interface.
  `class`
  `DataImport.ParseDate`
  Implements the translate method of the DataTranslator interface.
  `class`
  `DataImport.ParseDateTime`
  Implements the translate method of the DataTranslator interface.
  `class`
  `DataImport.ParseFloat`
  Implements the translate method of the DataTranslator interface.
  `class`
  `DataImport.ParseNumber`
  Implements the translate method of the DataTranslator interface.
  `class`
  `DataImport.ParseText`
  Implements the translate method of the DataTranslator interface.
  `class`
  `DataImport.ParseTime`
  Implements the translate method of the DataTranslator interface.

- ## Constructor Summary

  Constructors
  Constructor
  Description
  `DataImport()`
  Create a DataImport configured for CSV import with default quote string.
  `DataImport(DataImport.ImportFormat theInputType, String theDelimiter)`
  Create a DataImport configured for the specified import format.
  `DataImport(DataImport.ImportFormat theInputType, String theDelimiter, String theQuoteString)`
  Create a DataImport configured for the specified import format and quote string.
  `DataImport(String theInputType, String theDelimiter, String theQuoteString)`
  Create a DataImport configured for the specified import format and quote string.

- ## Method Summary

  Modifier and Type
  Method
  Description
  `Map`
  `importDataSourceRecord(Map record, DataSource dataSource)`
  See `importDataSourceRecord(Map, Map, DataSource)`
  `Map`
  `importDataSourceRecord(Map record, String dataSourceName)`
  See `importDataSourceRecord(Map, Map, String)`
  `Map`
  `importDataSourceRecord(Map record, List columns, String dataSourceName)`
  See `importDataSourceRecord(Map, Map, String)`
  `Map`
  `importDataSourceRecord(Map record, Map columnRemap, DataSource ds)`
  Imports provided record (as `Map` parameter) to dataSource record.
  `Map`
  `importDataSourceRecord(Map record, Map columnRemap, String dataSourceName)`
  Imports provided record (as `Map` parameter) to dataSource record.
  `List`
  `importDataSourceRecords(Reader in, String dataSourceName)`
  See `importDataSourceRecords(Reader, Map, Map, String)`
  `List`
  `importDataSourceRecords(Reader in, List columns, String dataSourceName)`
  See `importDataSourceRecords(Reader, Map, Map, String)`
  `List`
  `importDataSourceRecords(Reader in, List columns, Map translators, String dataSourceName)`
  See `importDataSourceRecords(Reader, Map, Map, String)`
  `List`
  `importDataSourceRecords(Reader in, Map columnRemap, String dataSourceName)`
  See `importDataSourceRecords(Reader, Map, Map, String)`
  `List`
  `importDataSourceRecords(Reader in, Map columnRemap, Map translators, String dataSourceName)`
  Import from `InputStream` in either CSV/TSV, JSON or XML format and return the imported records in a List.
  `long`
  `importToDataSource(Reader in, String dataSourceName)`
  See `importToDataSource(Reader, Map, Map, String)`
  `long`
  `importToDataSource(Reader in, List columns, String tableName)`
  See `importToDataSource(Reader, Map, Map, String)`
  `long`
  `importToDataSource(Reader in, List columns, Map translators, String tableName)`
  See `importToDataSource(Reader, Map, Map, String)`
  `long`
  `importToDataSource(Reader in, Map columnRemap, String tableName)`
  See `importToDataSource(Reader, Map, Map, String)`
  `long`
  `importToDataSource(Reader in, Map columnRemap, Map translators, String dataSourceName)`
  Import from `InputStream` in either CSV/TSV, JSON or XML format and save the imported records to the target SQLDataSource.
  `long`
  `importToDataSource(Map record, String dataSourceName)`
  See `importToDataSource(Map, Map, String)`
  `long`
  `importToDataSource(Map record, List columns, String dataSourceName)`
  See `importToDataSource(Map, Map, String)`
  `long`
  `importToDataSource(Map record, Map columnRemap, String dataSourceName)`
  Imports provided record (as `Map` parameter) to dataSource.
  `List`
  `importToRows(Reader in)`
  See `importToRows(Reader, Map, Map)`
  `List`
  `importToRows(Reader in, List columns)`
  See `importToRows(Reader, Map, Map)`
  `List`
  `importToRows(Reader in, List columns, Map translators)`
  See `importToRows(Reader, Map, Map)`
  `List`
  `importToRows(Reader in, Map columnRemap)`
  See `importToRows(Reader, Map, Map)`
  `List`
  `importToRows(Reader in, Map columnRemap, Map translators)`
  Import from `InputStream` in either CSV/TSV, JSON or XML format to a List of Maps.
  `void`
  `setAutoInterpretBooleans(boolean auto)`
  If true, the import process auto interprets boolean fields values converting them to `Booleans`, otherwise leaves them as `Strings`.
  `void`
  `setPopulateDisplayFields(boolean populate)`
  If true, the import process populates displayField values with the import data in cases where it transforms the underlying value using a related display record.

- ## Constructor Details

  - ### DataImport

    public DataImport()
    Create a DataImport configured for CSV import with default quote string.

  - ### DataImport

    public DataImport(DataImport.ImportFormat theInputType, String theDelimiter)
    Create a DataImport configured for the specified import format.
    Parameters:
      `theInputType` - the form of the data input.
      `theDelimiter` - a `java.lang.String` character used as the delimiter, by default, `","`

  - ### DataImport

    public DataImport(DataImport.ImportFormat theInputType, String theDelimiter, String theQuoteString)
    Create a DataImport configured for the specified import format and quote string.
    Parameters:
      `theInputType` - the form of the data input.
      `theDelimiter` - a `java.lang.String` character used as the delimiter, by default, `","`
      `theQuoteString` - for delimited input only, a `java.lang.String` (`"/""`, by default) to signify a double quote since a double quote would otherwise be misinterpreted as the end of a string by the parser

  - ### DataImport

    public DataImport(String theInputType, String theDelimiter, String theQuoteString)
    Create a DataImport configured for the specified import format and quote string.
    Parameters:
      `theInputType` - the form of the data input as a String
      `theDelimiter` - a `java.lang.String` character used as the delimiter, by default, `","`
      `theQuoteString` - for delimited input only, a `java.lang.String` (`"/""`, by default) to signify a double quote since a double quote would otherwise be misinterpreted as the end of a string by the parser

- ## Method Details

  - ### setPopulateDisplayFields

    public void setPopulateDisplayFields(boolean populate)
    If true, the import process populates displayField values with the import data in cases where it transforms the underlying value using a related display record. For example, if a field "countryId" has a `displayField` of "countryName". If this flag is set then as part of the process of importing a "countryId" value and transforming the import value "United States" into the corresponding id value, the importer also sets "countryName" on that record to the display value it just transformed (ie, "United States"). By default displayFields are not populated - you must pass true to this method if you require that behavior.

    See `DataSource.getRelatedDisplayRecord(String, Object, DSRequest)` for more details about related display records and import transformation.

    Parameters:
      `populate` - If true, populate displayFields with the import values used to derive a key via a related display record. If false, just leave displayFields unpopulated

  - ### setAutoInterpretBooleans

    public void setAutoInterpretBooleans(boolean auto)
    If true, the import process auto interprets boolean fields values converting them to `Booleans`, otherwise leaves them as `Strings`.

    Conversion rules:

    - Accept all caps permutations of "true"/"false", "yes"/"no", and "null"
    - "T" == true, "F" == false
    - "Y" == true, "N" == false
    - "0" is false
    - empty string is null
    - everything else is true
    Parameters:
      `auto` - If true, the import process auto interprets values boolean fields values converting them to `Booleans`, otherwise leaves them as `Strings`.

  - ### importToDataSource

    public long importToDataSource(Reader in, String dataSourceName) throws Exception
    See `importToDataSource(Reader, Map, Map, String)`
    Throws:
      `Exception`

  - ### importToDataSource

    public long importToDataSource(Reader in, List columns, String tableName) throws Exception
    See `importToDataSource(Reader, Map, Map, String)`
    Throws:
      `Exception`

  - ### importToDataSource

    public long importToDataSource(Reader in, Map columnRemap, String tableName) throws Exception
    See `importToDataSource(Reader, Map, Map, String)`
    Throws:
      `Exception`

  - ### importToDataSource

    public long importToDataSource(Reader in, List columns, Map translators, String tableName) throws Exception
    See `importToDataSource(Reader, Map, Map, String)`
    Throws:
      `Exception`

  - ### importToDataSource

    public long importToDataSource(Reader in, Map columnRemap, Map translators, String dataSourceName) throws Exception
    Import from `InputStream` in either CSV/TSV, JSON or XML format and save the imported records to the target SQLDataSource.

    Search for "testData" in SmartClient Reference for the description of supported formats and examples. Same rules apply here.

    An optional `columnRemap` can be provided to translate between column names in CSV/TSV or property names in JSON to the field names of the DataSource. Use `null` as a Map value to cause data for a column to be discarded.
    For signatures that take "List columns", this is the same as providing a columnRemap does not rename any input fields.

    If no `columnRemap` or incomplete `columnRemap` is provided, column names that are not re-mapped will be matched to DataSource fields by comparing to both the field name and field title, ignoring letter case. Any column name that isn't matched to a DataSource field is discarded.

    For delimited input, the header line may be omitted. `DataImport` will attempt to automatically detect whether the first line is a header by attempting to match the data to expected column names. If the header line is omitted, the columns will be assumed to be in the order of the `columnRemap` if provided (use a class such as `LinkedHashMap` to preserve order), or in the order of the DataSource fields if no `columnRemap` is provided. Any extra columns will be discarded.

    Any SimpleType that inherits from the given base type also uses the same translators: `DataImport.ParseDate`
    `DataImport.ParseDateTime`
    `DataImport.ParseTime`
    `DataImport.ParseText`
    `DataImport.ParseNumber`
    `DataImport.ParseFloat`
    `DataImport.ParseBoolean`

    For CSV/TSV and JSON data a set of translators can be provided to in the format expected by `importToRows`. By default, the following translations are applied based on DataSource field type:

    - `int`, `integer`, `sequence` or `number`: parsed as Java Integer
    - `float` or `decimal`: parsed as a Java Double
    - `date`: parsed as a date value by standard Java `DataFormat`, lenient parsing in current locale
    - `datetime`: parsed as a datetime value by standard Java `DataFormat`, lenient parsing in current locale
    Parameters:
      `in` - input file as a Reader
      `columnRemap` - an optional mapping between the input data and the field names of the target DataSource.
      `translators` - optional translators to use to transform the data from the input to the type the DataSourceField expects
      `dataSourceName` - the ID of the target DataSource
    Returns:
      `1` if the import was successful, and `0` otherwise
    Throws:
      `Exception`

  - ### importToDataSource

    public long importToDataSource(Map record, String dataSourceName) throws Exception
    See `importToDataSource(Map, Map, String)`
    Throws:
      `Exception`

  - ### importToDataSource

    public long importToDataSource(Map record, List columns, String dataSourceName) throws Exception
    See `importToDataSource(Map, Map, String)`
    Throws:
      `Exception`

  - ### importToDataSource

    public long importToDataSource(Map record, Map columnRemap, String dataSourceName) throws Exception
    Imports provided record (as `Map` parameter) to dataSource. This method uses same logic with column re-mapping as `importToDataSource(...)` and general features of `DataImport` such as transforming imported data according to import strategies.
    Parameters:
      `record` - input record as Map
      `columnRemap` - an optional mapping between the input data and the field names of the target DataSource.
      `dataSourceName` - the ID of the target DataSource
    Returns:
      `1` if the import was successful, and `0` otherwise
    Throws:
      `Exception`

  - ### importDataSourceRecords

    public List importDataSourceRecords(Reader in, String dataSourceName) throws Exception
    See `importDataSourceRecords(Reader, Map, Map, String)`
    Throws:
      `Exception`

  - ### importDataSourceRecords

    public List importDataSourceRecords(Reader in, List columns, String dataSourceName) throws Exception
    See `importDataSourceRecords(Reader, Map, Map, String)`
    Throws:
      `Exception`

  - ### importDataSourceRecords

    public List importDataSourceRecords(Reader in, Map columnRemap, String dataSourceName) throws Exception
    See `importDataSourceRecords(Reader, Map, Map, String)`
    Throws:
      `Exception`

  - ### importDataSourceRecords

    public List importDataSourceRecords(Reader in, List columns, Map translators, String dataSourceName) throws Exception
    See `importDataSourceRecords(Reader, Map, Map, String)`
    Throws:
      `Exception`

  - ### importDataSourceRecords

    public List importDataSourceRecords(Reader in, Map columnRemap, Map translators, String dataSourceName) throws Exception
    Import from `InputStream` in either CSV/TSV, JSON or XML format and return the imported records in a List. Note that imported records are not saved, for that consider using `importToDataSource(Reader, Map, Map, String)` API.

    Search for "testData" in SmartClient Reference for the description of supported formats and examples. Same rules apply here.

    An optional `columnRemap` can be provided to translate between column names in CSV/TSV or property names in JSON to the field names of the DataSource. Use `null` as a Map value to cause data for a column to be discarded.

    If no `columnRemap` or incomplete `columnRemap` is provided, column names that are not re-mapped will be matched to DataSource fields by comparing to both the field name and field title, ignoring letter case. Any column name that isn't matched to a DataSource field is discarded.

    For delimited input, the header line may be omitted. `DataImport` will attempt to automatically detect whether the first line is a header by attempting to match the data to expected column names. If the header line is omitted, the columns will be assumed to be in the order of the `columnRemap` if provided (use a class such as `LinkedHashMap` to preserve order), or in the order of the DataSource fields if no `columnRemap` is provided. Any extra columns will be discarded.

    Any SimpleType that inherits from the given base type also uses the same translators: `DataImport.ParseDate`
    `DataImport.ParseDateTime`
    `DataImport.ParseTime`
    `DataImport.ParseText`
    `DataImport.ParseNumber`
    `DataImport.ParseFloat`
    `DataImport.ParseBoolean`

    For CSV/TSV and JSON data a set of translators can be provided to in the format expected by `importToRows`. By default, the following translations are applied based on DataSource field type:

    - `int`, `integer`, `sequence` or `number`: parsed as Java Integer
    - `float` or `decimal`: parsed as a Java Double
    - `date`: parsed as a date value by standard Java `DataFormat`, lenient parsing in current locale
    - `datetime`: parsed as a datetime value by standard Java `DataFormat`, lenient parsing in current locale
    Parameters:
      `in` - input file as a Reader
      `columnRemap` - an optional mapping between the input data and the field names of the target DataSource.
      `translators` - optional translators to use to transform the data from the input to the type the DataSourceField expects
      `dataSourceName` - the ID of the target DataSource
    Returns:
      the imported records in a List
    Throws:
      `Exception`

  - ### importDataSourceRecord

    public Map importDataSourceRecord(Map record, String dataSourceName) throws Exception
    See `importDataSourceRecord(Map, Map, String)`
    Throws:
      `Exception`

  - ### importDataSourceRecord

    public Map importDataSourceRecord(Map record, DataSource dataSource) throws Exception
    See `importDataSourceRecord(Map, Map, DataSource)`
    Throws:
      `Exception`

  - ### importDataSourceRecord

    public Map importDataSourceRecord(Map record, List columns, String dataSourceName) throws Exception
    See `importDataSourceRecord(Map, Map, String)`
    Throws:
      `Exception`

  - ### importDataSourceRecord

    public Map importDataSourceRecord(Map record, Map columnRemap, String dataSourceName) throws Exception
    Imports provided record (as `Map` parameter) to dataSource record. This method uses same logic column re-mapping as `importDataSourceRecords(...)` and general features of `DataImport` such as transforming imported data according to import strategies.
    Parameters:
      `record` - input record as a Map
      `columnRemap` - an optional mapping between the input data and the field names of the target DataSource.
      `dataSourceName` - the ID of the target DataSource
    Returns:
      the imported record as Map
    Throws:
      `Exception`

  - ### importDataSourceRecord

    public Map importDataSourceRecord(Map record, Map columnRemap, DataSource ds) throws Exception
    Imports provided record (as `Map` parameter) to dataSource record. This method uses same logic column re-mapping as `importDataSourceRecords(...)` and general features of `DataImport` such as transforming imported data according to import strategies.
    Parameters:
      `record` - input record as a Map
      `columnRemap` - an optional mapping between the input data and the field names of the target DataSource.
      `ds` - instance of the target DataSource
    Returns:
      the imported record as Map
    Throws:
      `Exception`

  - ### importToRows

    public List importToRows(Reader in) throws Exception
    See `importToRows(Reader, Map, Map)`
    Throws:
      `Exception`

  - ### importToRows

    public List importToRows(Reader in, List columns) throws Exception
    See `importToRows(Reader, Map, Map)`
    Throws:
      `Exception`

  - ### importToRows

    public List importToRows(Reader in, Map columnRemap) throws Exception
    See `importToRows(Reader, Map, Map)`
    Throws:
      `Exception`

  - ### importToRows

    public List importToRows(Reader in, List columns, Map translators) throws Exception
    See `importToRows(Reader, Map, Map)`
    Throws:
      `Exception`

  - ### importToRows

    public List importToRows(Reader in, Map columnRemap, Map translators) throws Exception
    Import from `InputStream` in either CSV/TSV, JSON or XML format to a List of Maps.

    Search for "testData" in SmartClient Reference for the description of supported formats and examples. Same rules apply here.

    This API essentially performs part of the steps of `importToDataSource()` but does not actually insert records into a DataSource, instead just returning a List of Maps which could then be inserted into a DataSource, validated further, serialized to XML or otherwise processed.

    Input format and delimiter is specified in the constructor or via setInputType(). CSV/TSV delimited data is expected to have a leading line providing column names, which will be used as keys for the Maps returned.

    An optional `columnRemap` can be provided to translate between column names in CSV/TSV or property names in JSON to a different set of column names to be used in the output. Use a `null` as the value of a key in this Map to indicate that data for the column should be discarded. The matching process against the keys of the columnRemap is case-insensitive.
    For signatures that take "List columns", this is the same as providing a columnRemap does not rename any input fields.

    An optional set of translators can be provided to translate to desired Java types. The `translators` Map should be a map from column name (in the CSV/TSV file - not the remapped column name) to the fully qualified name of a specific inner class that will do the translation (for example: "com.isomorphic.tools.DataImport.ParseText"). The class invokes its own translate method which takes any Java Object and produces any Java Object as output.

    If the input type is set to autoDetect, `importToRows()` will attempt to autodetect the input format. In addition, for delimited input, if the delimiter is unspecified, `importToRows()` will attempt to detect whether the delimiter is a comma (",") or tab ("\t"), and will throw an exception if auto-detection fails.

    Example usage for CSV parsing (which is the default input type and delimiter):

         InputStream is = ServletTools.loadWebRootFile("files/portfolios.csv");
         Reader reader = new InputStreamReader(is);
         List portfolios = new DataImport().importToRows(reader);

    Parameters:
      `in` - a character-stream reader set to read the data input stream
      `columnRemap` - optional mapping from input column/property names to output Map keys.
      `translators` - optional list of translators
    Returns:
      imported rows as a List of Maps
    Throws:
      `Exception`