Monitoring

Track job status, view logs, and monitor operations in real time.

Jobs Dashboard

The Monitoring page in the sidebar shows all jobs across your pipelines. Use it to get a high-level view of what's running, what's failed, and what's completed.

Status Cards

At the top of the dashboard, status cards show counts for Running, Pending, Completed, and Failed jobs. Click a card to filter the table below.

Filters

Filter jobs by status, pipeline, date range, or job type (source sync vs. destination export). The table updates in real time.

Job Detail

Click any job to see its full detail: start/end times, duration, rows processed, error messages (if failed), and associated pipeline and connection.

Job Statuses

Every job passes through one of these statuses:

PENDING

Job is queued and waiting to be picked up by a worker.

RUNNING

Job is actively syncing or exporting data.

WAITING_FOR_BULK

A Shopify bulk operation has been submitted and the job is waiting for Shopify to finish processing. The scheduler polls periodically and resumes when ready.

COMPLETED

Job finished successfully. Check the job detail for row counts and duration.

FAILED

Job encountered an error. Check the error message in the job detail for troubleshooting.

CANCELLED

Job was manually cancelled by a user before completion.

Logs

The log viewer provides detailed runtime output for each job. Access it from the job detail page or from the Monitoring section.

Filtering

Filter logs by level (info, warn, error), source (plugin name or system), date range, and free-text search. Results are paginated for large log sets.

Export

Export logs as CSV or JSON for external analysis or archival.

Shopify Bulk Operations

For Shopify sources, SnowPipe uses GraphQL Bulk Operations to efficiently sync large catalogs. The monitoring dashboard includes a dedicated view for tracking these operations.

Operation Phases

Initializing→Fetching→Waiting→Downloading→Finalizing

During the Waiting phase, Shopify is processing the bulk query. SnowPipe's scheduler polls automatically and resumes the job when the data is ready to download.

Concurrent Operations

SnowPipe can run 1–5 concurrent bulk operations per Shopify store, automatically segmenting large date ranges for parallel processing. This is handled by the Bulk Operation Manager.

Cancellation

You can cancel a running bulk operation from the monitoring view. This sends a cancellation request to Shopify and marks the associated job as CANCELLED.

Pipeline-Level Monitoring

Each pipeline has its own Jobs tab showing job history for that specific pipeline.

Success rate — Percentage of jobs that completed successfully.
Records imported — Total rows fetched from the source in the last run.
Records exported — Total rows written to the destination in the last run.
Duration trends — Compare run times across recent jobs to spot slowdowns.

Need help?

Reach out at support@snowforge.dev