Repository: ClickHouse/ClickHouse
File: src/Storages/ObjectStorage/StorageObjectStorage.cpp (C++)

Code before:
    metadata.setComment(comment);

    /// I am not sure this is actually required, but just in case
    if (configuration->partition_strategy)
    {
        metadata.partition_key = configuration->partition_strategy->getPartitionKeyDescription();
    }

    setVirtuals(VirtualColumnUtils::getVirtualsForFileLikeStorage(
        metadata.columns,
        context,
        format_settings,
        configuration->partition_strategy_type,
        sample_path));

    setInMemoryMetadata(metadata);

    /// This will update metadata for table function which contains specific information about table
    /// state (e.g. for Iceberg). It is done because select queries for table functions are executed
    /// in a different way and clickhouse can execute without calling updateExternalDynamicMetadataIfExists.
    if (!do_lazy_init && is_table_function && configuration->needsUpdateForSchemaConsistency())
    {
        auto metadata_snapshot = configuration->getStorageSnapshotMetadata(context);
        setInMemoryMetadata(metadata_snapshot);
    }
    if (!configuration->isDataLakeConfiguration() && !columns_in_table_or_function_definition.empty())
    {
        String sample_path_schema;
        std::optional<ColumnsDescription> schema_file;
        try
        {
            schema_file = resolveSchemaFromData(object_storage_, configuration_, {}, sample_path_schema, context);
        }
        catch (...)
        {
            tryLogCurrentException(log);
        }
        if (schema_file)
        {
            for (const auto & column : columns_in_table_or_function_definition)
                if (!schema_file->tryGet(column.name))
                    throw Exception(ErrorCodes::BAD_ARGUMENTS, "Cannot find column in schema {}", column.name);
        }
    }
}

String StorageObjectStorage::getName() const
{
    return configuration->getEngineName();
}


Review comment (bug):
> !configuration->isDataLakeConfiguration()

Why do we exclude data lakes?

> !columns_in_table_or_function_definition.empty()

In case of table function, if I am not mistaken, this will also contain columns not just specified in function arguments, but resolved from remote file in `getActualTableStructure` when `structure == auto`. In this case we should not do this check as it will already be consistent. Can you please check if this is indeed the case?

Also this schema check is not lightweight (given that it reads from object storage), so it contradicts with `do_lazy_init` purpose (to not do anything heavy in constructor). Though as I remember this is only used in `DatabaseDataLake`, but would be good to recheck.

Also IMO throwing `Cannot find column in schema` in case of table engine (not table function) should be done only in case of CREATE query, while ATTACH should work -- to not fail server startup.

Code after:
    ///    in the metadata, not present in parquet file. Like hive partitioning, but in metadata
    ///    files instead of path. Currently these columns are added to the block outside parquet
    ///    reader. If they appear in prewhere expression, parquet reader gets a "no column in block"
    ///    error. Unlike hive partitioning, we can't (?) just return these columns from
    ///    supportedPrewhereColumns() because at the time of the call the delta lake metadata hasn't
    ///    been read yet. So we should probably pass these columns to the parquet reader instead of
    ///    adding them outside.
    ///  * There's a bug in StorageObjectStorageSource::createReader: it makes a copy of
    ///    FormatFilterInfo, but for some reason unsets prewhere_info and row_level_filter_info.
    ///    There's probably no reason for this, and it should just copy those fields like the others.
    ///  * If the table contains files in different formats, with only some of them supporting
    ///    prewhere, things break.
    supports_prewhere = !configuration->isDataLakeConfiguration() && format_supports_prewhere;
    supports_tuple_elements = format_supports_prewhere;

    StorageInMemoryMetadata metadata;
    metadata.setColumns(columns);
    metadata.setConstraints(constraints_);
    metadata.setComment(comment);

    /// I am not sure this is actually required, but just in case
    if (configuration->partition_strategy)
    {
        metadata.partition_key = configuration->partition_strategy->getPartitionKeyDescription();
    }

    setVirtuals(VirtualColumnUtils::getVirtualsForFileLikeStorage(
        metadata.columns,
        context,
        format_settings,
        configuration->partition_strategy_type,
        sample_path));

    setInMemoryMetadata(metadata);

    /// This will update metadata for table function which contains specific information about table
    /// state (e.g. for Iceberg). It is done because select queries for table functions are executed
    /// in a different way and clickhouse can execute without calling updateExternalDynamicMetadataIfExists.
    if (!do_lazy_init && is_table_function && configuration->needsUpdateForSchemaConsistency())
    {
        auto metadata_snapshot = configuration->getStorageSnapshotMetadata(context);
        setInMemoryMetadata(metadata_snapshot);
    }

}

String StorageObjectStorage::getName() const
{
    return configuration->getEngineName();
}