PHP Generators Streaming Large Data Sets

By Khimananda Oli | Last reviewed: August 2026

Processing millions of records often crashes PHP applications because standard arrays load everything into memory at once. PHP generators streaming large data sets solve this fatal error by yielding one value at a time, keeping memory usage constant regardless of dataset size. This guide shows you exactly how to replace memory-hungry loops with efficient generators in production environments.

How do PHP generators streaming large data sets reduce memory usage?

When you iterate over a standard array containing one million records, PHP allocates memory for every single element before your loop executes even once. On a typical VPS configured via our Ubuntu PHP setup guide, hitting the 128MB or 256MB memory limit triggers an immediate fatal error. Generators fundamentally change this allocation model by implementing the iterator interface internally without materializing the full collection.

The critical distinction lies in execution state. A generator function pauses at each yield statement, returning control to the caller while preserving its local variables. When the caller requests the next value, execution resumes exactly where it left off. This suspend-resume cycle means only the current record occupies memory, not the entire history of processed items.

Measuring the actual memory difference

You should always verify memory claims empirically. Use memory_get_peak_usage(true) before and after processing to capture real allocation:

<?php
// Standard approach - loads everything first
$records = range(1, 1000000);
foreach ($records as $record) {
    // process
}
echo "Array peak: " . memory_get_peak_usage(true) / 1024 / 1024 . " MB\n";

// Generator approach - yields on demand
function generateRecords(int $count): Generator {
    for ($i = 1; $i <= $count; $i++) {
        yield $i;
    }
}

foreach (generateRecords(1000000) as $record) {
    // process
}
echo "Generator peak: " . memory_get_peak_usage(true) / 1024 / 1024 . " MB\n";

In practice on PHP 8.4, the array version consumes approximately 38MB for integers alone, while the generator version stays under 1MB regardless of count. For complex objects or associative arrays representing database rows, this gap widens to orders of magnitude.

How do you stream large CSV files with PHP generators?

CSV imports are the most common use case for PHP generators streaming large data sets. Financial reports, e-commerce product catalogs, and legacy system exports frequently arrive as multi-gigabyte CSV files that cannot fit in memory. The pattern combines fgetcsv() with yield to create a lazy file reader.

<?php
function streamCsv(string $filePath, bool $skipHeader = true): Generator {
    $handle = fopen($filePath, 'r');
    if ($handle === false) {
        throw new RuntimeException("Cannot open file: {$filePath}");
    }
    
    try {
        if ($skipHeader) {
            fgetcsv($handle); // Skip header row
        }
        
        while (($row = fgetcsv($handle)) !== false) {
            yield $row;
        }
    } finally {
        fclose($handle); // Always close, even on exception
    }
}

// Usage: process 2GB file with ~2KB memory
foreach (streamCsv('/data/exports/transactions.csv') as $row) {
    transformAndSave($row);
}

The finally block is non-negotiable here. Without it, an exception mid-stream leaves the file handle open, eventually exhausting your OS-level file descriptor limit. I have debugged this exact issue in production systems where error handling was added later without revisiting resource cleanup.

Handling malformed CSV rows gracefully

Real-world CSV files contain encoding issues, inconsistent delimiters, and truncated lines. Wrap the yield in validation logic rather than letting bad data crash the entire import:

function streamValidatedCsv(string $path, int $expectedColumns): Generator {
    $handle = fopen($path, 'r');
    $lineNumber = 1;
    
    try {
        while (($row = fgetcsv($handle)) !== false) {
            $lineNumber++;
            if (count($row) !== $expectedColumns) {
                error_log("Skipping malformed line {$lineNumber}");
                continue;
            }
            yield $lineNumber => $row;
        }
    } finally {
        fclose($handle);
    }
}

This approach logs problems without halting the pipeline. For audit-sensitive workloads like financial reconciliation, pair this with the structured logging patterns described in our structured logging best practices guide to maintain compliance trails during batch processing.

How do you integrate PHP generators with database cursors?

Databases already support cursor-based fetching, but ORM abstractions often hide this behind eager-loading methods. When exporting user data or generating reports from tables with millions of rows, bypass the ORM's collection hydration and stream directly from the database cursor using PHP generators streaming large data sets.

PDO unbuffered query pattern

MySQL buffers entire result sets client-side by default. Disable this with MYSQL_ATTR_USE_BUFFERED_QUERY to enable true server-side cursors:

<?php
function streamUsers(PDO $pdo): Generator {
    $stmt = $pdo->prepare('SELECT id, email, created_at FROM users');
    $stmt->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false);
    $stmt->execute();
    
    while ($row = $stmt->fetch(PDO::FETCH_ASSOC)) {
        yield $row;
    }
}

// Process without loading all users into memory
foreach (streamUsers($pdo) as $user) {
    sendWelcomeEmail($user['email']);
}

A common mistake is running additional queries inside the generator loop while an unbuffered result set is active. MySQL prohibits this on the same connection. Either use a separate connection for inner queries or fetch the current row into a variable before yielding. For high-throughput ETL pipelines, consider the replication topology discussed in our MySQL master-slave replication setup article to isolate read-heavy streaming from write operations.

Laravel Lazy Collections

If you work within Laravel, the framework wraps generator logic in LazyCollection. This provides chainable methods like filter(), map(), and chunk() while preserving lazy evaluation:

// Laravel: streams automatically, never loads full table
User::cursor()->filter(fn($u) => $u->is_active)
    ->each(function ($user) {
        Notification::send($user, new ReportReady());
    });

Beware of accidentally breaking laziness by calling ->all(), ->toArray(), or ->count() on a lazy collection. These force full materialization and defeat the purpose entirely. Profile with DB::enableQueryLog() during development to confirm streaming behavior.

What are the performance trade-offs of PHP generators versus arrays?

Generators optimize memory at the cost of CPU overhead and lost random access. Understanding these trade-offs prevents misapplication in latency-sensitive paths where arrays remain superior.

The iteration overhead comes from generator object creation, state suspension, and resumption on each next() call. Benchmarks on PHP 8.4 show roughly 15–25% slower traversal compared to native array iteration. However, when the alternative is swapping to disk or crashing entirely, this CPU cost is irrelevant. Generators win whenever dataset size approaches or exceeds available RAM.

When NOT to use generators

• Small datasets under 10K records: Array overhead is negligible; generator complexity adds no value.
• Multiple passes required: Sorting, grouping, or comparing elements requires re-reading. Generators cannot rewind.
• Random access needed: Binary search, index lookups, or pagination by offset require materialized structures.
• CPU-bound transformations: If processing per-item dominates runtime, generator overhead compounds. Profile first.

How do you handle errors and backpressure in generator pipelines?

Generators decouple production from consumption, which introduces subtle failure modes. Unlike arrays where validation happens upfront, generator errors surface mid-iteration when recovery is harder.

Exception safety inside generators

Exceptions thrown inside a generator propagate to the consumer's foreach or ->next() call. However, cleanup code after the failing yield never executes unless wrapped in finally. Always protect resources:

function safeStream(string $path): Generator {
    $fp = fopen($path, 'r');
    try {
        while ($line = fgets($fp)) {
            $parsed = json_decode($line, true);
            if (json_last_error() !== JSON_ERROR_NONE) {
                throw new InvalidArgumentException("Invalid JSON at line");
            }
            yield $parsed;
        }
    } finally {
        fclose($fp); // Guaranteed execution
    }
}

Implementing backpressure for external APIs

When streaming data to rate-limited APIs or slow databases, the consumer controls pace naturally since generators only produce when pulled. Add explicit throttling when downstream systems signal overload:

foreach (streamRecords() as $record) {
    $response = $api->post($record);
    if ($response->getStatusCode() === 429) {
        $retryAfter = $response->getHeaderLine('Retry-After');
        sleep((int) $retryAfter ?: 5);
    }
}

This pull-based flow control is inherent to generator design. The producer never races ahead of the consumer, eliminating buffer bloat that plagues queue-based architectures. For observability into pipeline throughput and error rates, instrument your generators following the monitoring patterns in our Prometheus metrics monitoring fundamentals guide.

Implementing PHP Generators Streaming Large Data Sets in Production

Adopting PHP generators streaming large data sets transforms fragile batch jobs into resilient, memory-safe pipelines. Start by identifying your highest-risk endpoints: CSV exports, admin list views without pagination, and background report generation. Replace array-based implementations incrementally, profiling memory before and after each change. Remember that generators are single-pass forward-only streams; design your business logic accordingly. When applied correctly, this pattern eliminates an entire class of production outages related to memory exhaustion while keeping your infrastructure costs predictable. If you need help auditing your PHP application's memory profile or designing streaming pipelines for compliance-sensitive workloads, reach out to discuss your specific architecture.