Processing a 2 GB CSV in PHP Without Running Out of Memory

The client's export was 2.3 GB of order history, and the import script greeted it with the classic: Allowed memory size of 536870912 bytes exhausted. The script's crime was one line - file($path) - which loads every row into an array before the loop even starts.

Generators flip the model

A function containing yield doesn't run when called - it returns a Generator that produces values on demand, one per loop iteration, remembering its position between values:

function readCsv(string $path): Generator
{
    $handle = fopen($path, 'r');
    $headers = fgetcsv($handle);

    while (($row = fgetcsv($handle)) !== false) {
        yield array_combine($headers, $row);
    }

    fclose($handle);
}

foreach (readCsv('orders-2015-2026.csv') as $order) {
    importOrder($order);   // one row in memory at a time
}

Memory use is flat regardless of file size - the 2.3 GB file processes in the same few megabytes as a 2 KB one. The foreach looks identical to iterating an array; the caller doesn't need to know or care that it's lazy.

Pipelines: where generators get elegant

Generators compose. Filter and transform steps each take an iterable and yield results, so you can chain them like Unix pipes:

function onlyPaid(iterable $orders): Generator
{
    foreach ($orders as $order) {
        if ($order['status'] === 'paid') yield $order;
    }
}

function inBatches(iterable $items, int $size): Generator
{
    $batch = [];
    foreach ($items as $item) {
        $batch[] = $item;
        if (count($batch) === $size) { yield $batch; $batch = []; }
    }
    if ($batch) yield $batch;
}

foreach (inBatches(onlyPaid(readCsv($path)), 500) as $batch) {
    $pdo->beginTransaction();
    foreach ($batch as $order) insertOrder($pdo, $order);
    $pdo->commit();   // batched transactions: the other half of import speed
}

Three reusable functions, still constant memory, and the batching gives you the transaction-per-500-rows pattern that makes bulk inserts fast.

Where else this applies

• Database exports: loop a PDOStatement directly (it's already iterable row-by-row) instead of fetchAll(), and stream rows out with fputcsv('php://output', ...). Millions of rows, no memory ceiling on either side.
• API pagination: a generator that fetches page after page internally but yields individual items gives callers a clean flat loop over "all items".
• yield from delegates to inner generators/arrays - handy for walking directory trees recursively.

One honest caveat: generators are forward-only and single-pass. No count(), no rewinding, no random access - if you need those, you needed an array anyway. For the "read a lot, touch each item once" shape that dominates data work, generators are the difference between scripts that scale and scripts that fall over. If your imports run through the browser, pair this with post-response processing so users aren't staring at a spinner while rows stream.