# `SparkEx.Column`
[🔗](https://github.com/lukaszsamson/spark_ex/blob/v0.1.1/lib/spark_ex/column.ex#L1)

Expression wrapper for Spark DataFrame columns.

A `Column` wraps an internal expression representation that gets encoded
into Spark Connect protobuf `Expression` messages by the `PlanEncoder`.

Columns are created via `SparkEx.Functions` constructors (`col/1`, `lit/1`, `expr/1`)
and combined using the operations defined here.

## Examples

    import SparkEx.Functions, only: [col: 1, lit: 1]

    col("age") |> SparkEx.Column.gt(lit(18))
    col("name") |> SparkEx.Column.alias_("user_name")
    col("score") |> SparkEx.Column.desc()

# `expr`

```elixir
@type expr() ::
  {:col, String.t()}
  | {:col, String.t(), term()}
  | {:lit, term()}
  | {:expr, String.t()}
  | {:col_regex, String.t()}
  | {:col_regex, String.t(), term()}
  | {:metadata_col, String.t()}
  | {:metadata_col, String.t(), term()}
  | {:fn, String.t(), [expr()], boolean()}
  | {:alias, expr(), String.t()}
  | {:alias, expr(), String.t(), String.t()}
  | {:sort_order, expr(), :asc | :desc, :nulls_first | :nulls_last | nil}
  | {:cast, expr(), String.t() | SparkEx.Types.data_type_proto()}
  | {:cast, expr(), String.t() | SparkEx.Types.data_type_proto(), :try}
  | {:star}
  | {:star, String.t()}
  | {:star, String.t() | nil, term()}
  | {:outer, expr()}
  | {:window, expr(), [expr()], [expr()], term()}
  | {:unresolved_extract_value, expr(), expr()}
  | {:update_fields, expr(), String.t(), expr() | nil}
  | {:lambda, expr(), [{:lambda_var, String.t()}]}
  | {:lambda_var, String.t()}
  | {:named_arg, String.t(), expr()}
  | {:call_function, String.t(), [expr()]}
  | {:subquery, atom(), term(), keyword()}
```

# `t`

```elixir
@type t() :: %SparkEx.Column{expr: expr()}
```

# `alias_`

```elixir
@spec alias_(t(), String.t(), keyword()) :: t()
```

Assigns an alias (name) to this column expression.

Optionally accepts a `metadata` keyword with a JSON-serializable map.

# `and_`

```elixir
@spec and_(t(), t() | term()) :: t()
```

Logical AND.

# `asc`

```elixir
@spec asc(t()) :: t()
```

Sort ascending (nulls first by default)

# `asc_nulls_first`

```elixir
@spec asc_nulls_first(t()) :: t()
```

Sort ascending with nulls first

# `asc_nulls_last`

```elixir
@spec asc_nulls_last(t()) :: t()
```

Sort ascending with nulls last

# `astype`

```elixir
@spec astype(t(), String.t() | SparkEx.Types.data_type_proto()) :: t()
```

Alias for `cast/2`.

# `between`

```elixir
@spec between(t(), term(), term()) :: t()
```

Returns true if the column value is between lower and upper (inclusive).

# `bitwise_and`

```elixir
@spec bitwise_and(t(), t() | term()) :: t()
```

Bitwise AND.

# `bitwise_not`

```elixir
@spec bitwise_not(t()) :: t()
```

Bitwise NOT.

# `bitwise_or`

```elixir
@spec bitwise_or(t(), t() | term()) :: t()
```

Bitwise OR.

# `bitwise_xor`

```elixir
@spec bitwise_xor(t(), t() | term()) :: t()
```

Bitwise XOR.

# `cast`

```elixir
@spec cast(t(), String.t() | SparkEx.Types.data_type_proto()) :: t()
```

Casts the column to the given type.

The type can be a Spark SQL type string (e.g. `"int"`, `"string"`, `"double"`)
or a Spark Connect DataType protobuf struct.

# `contains`

```elixir
@spec contains(t(), t() | term()) :: t()
```

String contains.

# `desc`

```elixir
@spec desc(t()) :: t()
```

Sort descending (nulls last by default)

# `desc_nulls_first`

```elixir
@spec desc_nulls_first(t()) :: t()
```

Sort descending with nulls first

# `desc_nulls_last`

```elixir
@spec desc_nulls_last(t()) :: t()
```

Sort descending with nulls last

# `divide`

```elixir
@spec divide(t(), t() | term()) :: t()
```

Division: `col / other`.

# `drop_fields`

```elixir
@spec drop_fields(t(), [String.t()]) :: t()
```

Drops fields from a struct column.

# `ends_with`

```elixir
@spec ends_with(t(), t() | term()) :: t()
```

String ends with.

# `endswith`

```elixir
@spec endswith(t(), t() | term()) :: t()
```

Alias for `ends_with/2`.

# `eq`

```elixir
@spec eq(t(), t() | term()) :: t()
```

Equality: `col == other`.

# `eq_null_safe`

```elixir
@spec eq_null_safe(t(), t() | term()) :: t()
```

Null-safe equality.

# `get_field`

```elixir
@spec get_field(t(), String.t()) :: t()
```

Extracts a field from a struct column by name.

# `get_item`

```elixir
@spec get_item(t(), t() | term()) :: t()
```

Extracts a value from an array by index or from a map by key.

# `gt`

```elixir
@spec gt(t(), t() | term()) :: t()
```

Greater than: `col > other`.

# `gte`

```elixir
@spec gte(t(), t() | term()) :: t()
```

Greater than or equal: `col >= other`.

# `ilike`

```elixir
@spec ilike(t(), t() | term()) :: t()
```

Case-insensitive LIKE.

# `is_nan`

```elixir
@spec is_nan(t()) :: t()
```

Returns true if the column is NaN.

# `is_not_null`

```elixir
@spec is_not_null(t()) :: t()
```

Returns true if the column is not null.

# `is_null`

```elixir
@spec is_null(t()) :: t()
```

Returns true if the column is null.

# `isin`

```elixir
@spec isin(t(), [term()] | SparkEx.DataFrame.t()) :: t()
```

Returns true if the column value is in the given list of values or subquery DataFrame.

# `like`

```elixir
@spec like(t(), t() | term()) :: t()
```

SQL LIKE pattern match.

# `lt`

```elixir
@spec lt(t(), t() | term()) :: t()
```

Less than: `col < other`.

# `lte`

```elixir
@spec lte(t(), t() | term()) :: t()
```

Less than or equal: `col <= other`.

# `minus`

```elixir
@spec minus(t(), t() | term()) :: t()
```

Subtraction: `col - other`.

# `mod`

```elixir
@spec mod(t(), t() | term()) :: t()
```

Modulo.

# `multiply`

```elixir
@spec multiply(t(), t() | term()) :: t()
```

Multiplication: `col * other`.

# `name`

```elixir
@spec name(t(), String.t()) :: t()
```

Alias for `alias_/2`.

# `negate`

```elixir
@spec negate(t()) :: t()
```

Unary negation.

# `neq`

```elixir
@spec neq(t(), t() | term()) :: t()
```

Not equal: `col != other`. Encodes as `not(==(a, b))` matching PySpark.

# `not_`

```elixir
@spec not_(t()) :: t()
```

Logical NOT.

# `or_`

```elixir
@spec or_(t(), t() | term()) :: t()
```

Logical OR.

# `otherwise`

```elixir
@spec otherwise(t(), t() | term()) :: t()
```

Adds a fallback value to a `when/2` expression chain.

# `outer`

```elixir
@spec outer(t()) :: t()
```

Marks this column for lateral join / generator context.

# `over`

```elixir
@spec over(t(), SparkEx.WindowSpec.t()) :: t()
```

Defines a window specification for this column expression.

## Examples

    import SparkEx.Functions, only: [col: 1]

    w = SparkEx.Window.partition_by(["dept"]) |> SparkEx.WindowSpec.order_by(["salary"])
    col("salary") |> SparkEx.Functions.row_number() |> SparkEx.Column.over(w)

# `plus`

```elixir
@spec plus(t(), t() | term()) :: t()
```

Addition: `col + other`.

# `pow`

```elixir
@spec pow(t(), t() | term()) :: t()
```

Computes `col` raised to the given power.

# `power`

```elixir
@spec power(t(), t() | term()) :: t()
```

Alias for `pow/2`.

# `rlike`

```elixir
@spec rlike(t(), t() | term()) :: t()
```

Regex pattern match.

# `starts_with`

```elixir
@spec starts_with(t(), t() | term()) :: t()
```

String starts with.

# `startswith`

```elixir
@spec startswith(t(), t() | term()) :: t()
```

Alias for `starts_with/2`.

# `substr`

```elixir
@spec substr(t(), t() | integer(), t() | integer()) :: t()
```

Returns a substring starting at `pos` for `len` characters.

# `transform`

```elixir
@spec transform(t(), (t() -&gt; t())) :: t()
```

Applies a transformation function to this column.

Equivalent to PySpark's `Column.transform(f)` which delegates to
the `transform` SQL function.

## Examples

    col("arr") |> Column.transform(fn x -> Column.plus(x, lit(1)) end)

# `try_cast`

```elixir
@spec try_cast(t(), String.t() | SparkEx.Types.data_type_proto()) :: t()
```

Try-casts the column to the given type. Returns null on cast failure instead of error.

# `when_`

```elixir
@spec when_(t(), t() | term()) :: t()
```

Creates an initial `when` branch from a condition column.

# `when_`

```elixir
@spec when_(t(), t(), t() | term()) :: t()
```

Appends another condition/value branch to an existing `when` chain.

# `with_field`

```elixir
@spec with_field(t(), String.t(), t() | term()) :: t()
```

Adds or replaces a field in a struct column.

---

*Consult [api-reference.md](api-reference.md) for complete listing*