Amazon OpenSearch Service introduces intelligent ingestion for logs and documents in Amazon S3

Amazon OpenSearch Serverless now provides intelligent ingestion, a managed feature that converts logs and documents stored in Amazon S3 into searchable content within Amazon OpenSearch Serverless (NextGen) collections. Users can initiate ingestion through the AWS Management Console with a few clicks or via a single API call, eliminating the need to manage ingestion infrastructure. The feature supports observability logs and structured data in JSON, NDJSON, and JSON array formats, as well as unstructured files in PDF, Word, plain text, and Markdown formats. Once configured, intelligent ingestion automatically detects data types, parses documents, and indexes the results without requiring manual setup or capacity planning.

Intelligent ingestion handles both one-time data loads, such as migrations or backfills, and continuous synchronization through ingestion streams to keep indices updated as new objects are added to S3 buckets. The feature scales capacity dynamically based on workload demands, up to the configured collection limit, and charges apply only for active processing time. Source S3 buckets can reside in the same AWS account as the collection or in a different account, offering flexibility in data source management.

For structured data and logs, intelligent ingestion applies transformations and optionally generates vector embeddings without requiring pipeline code. Unstructured data, including PDF, Word, plain text, and Markdown files, is parsed while preserving document structure such as headings, reading order, and tables. This ensures content is indexed in a format ready for semantic and hybrid search queries. Additionally, the feature processes visual elements like charts, diagrams, screenshots, and photos by generating text descriptions using a vision-language model, making visual information searchable alongside textual content.

Intelligent ingestion is available today for Amazon OpenSearch Serverless (NextGen) collections with Amazon S3 as the source, operating in seven AWS Regions. Users can access the feature through the Amazon OpenSearch Service console by selecting a collection and navigating to the Ingestion tab. Detailed guidance is provided in the Amazon OpenSearch Service Developer Guide, while pricing details are outlined in the Amazon OpenSearch Service pricing page.