An AI agent can orchestrate Polars data pipelines and launch SageMaker jobs automatically.
1. Install dependencies → pip install openai polars boto3.
2. Write a Polars script, e.g.:
import polars as pl
df = pl.read_csv('data.csv')
result = df.filter(pl.col('value') > 10).groupby('category').agg(pl.mean('value'))
result.write_csv('out.csv')3. Define an OpenAI function tool that runs the script:
import subprocess
def run_polars():
subprocess.run(['python', 'polars_job.py'], check=True)4. Create the agent and invoke the tool:
from openai import OpenAI
client = OpenAI()
response = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role':'user','content':'Run the data cleaning job'}],
functions=[{'name':'run_polars','description':'Execute Polars script'}]
)5. In SageMaker, wrap the script as a processing step and let the agent trigger it via Boto3.
Gotcha: Ensure the agent's runtime environment has access to the data files and proper IAM permissions, otherwise the job will fail.