Lambda(node.js)からS3のCSVオブジェクトをSelectObjectContentCommandで取得する – AWS SDK for JavaScript v3

Lambda(node.js)からS3のCSVオブジェクトをSelectObjectContentCommandで取得する – AWS SDK for JavaScript v3

Lambda(node.js)でs3 selectが使えるようになっているようです。s3 selectが使えるとcsvのデータを取得時に加工することができるようになります。

Introducing support for Amazon S3 Select in the AWS SDK for JavaScript | Amazon Web Services
We’re excited to announce support for the Amazon Simple Storage Service (Amazon S3) selectObjectContent API with event streams in the AWS SDK for JavaScript. Us...





npm i @aws-sdk/client-s3 csv-parse


s3のCSVオブジェクト取得してs3 selectして、csvモジュールで配列にします。

import {S3Client, SelectObjectContentCommand} from '@aws-sdk/client-s3'
import {parse} from 'csv-parse/sync'
export async function handler(event, context) {
  const input = {
    Bucket: 'バケット名',
    Key: 'tmp/testdata.csv',
    ExpressionType: 'SQL',
    Expression: 'SELECT s._1,s._2 FROM S3Object s', // 1,2列目のみ取得するSQL
    InputSerialization: {
      CSV: {
        FileHeaderInfo: 'NONE', // header行なし
        RecordDelimiter: '\r\n',
        FieldDelimiter: ',' // カンマ区切り
    OutputSerialization: {
      CSV: {}
  const client = new S3Client({
    region: 'ap-northeast-1'
  const command = new SelectObjectContentCommand(input)
  const data = await client.send(command)
  const csv = parse(await streamToCsv(data.Payload))
  console.log(csv) // [ [ 'aaa', 'bbb' ], [ 'ddd', 'eee' ] ]
  const response = {
    statusCode: 200,
    body: JSON.stringify('Hello from Lambda!'),
  return response

async function streamToCsv(generator) {
  const chunks = []
  for await (const value of generator) {
    if (value.Records) {
  return Buffer.concat(chunks).toString('utf8')


InvalidTextEncoding: UTF-8 encoding is required.

UTF-8のファイルのみ対応されているようで、SJISだと「InvalidTextEncoding: UTF-8 encoding is required.」エラーが発生します。

UTF-8 – UTF-8 is the only encoding type Amazon S3 Select supports.

SelectObjectContent - Amazon Simple Storage Service
This action filters the contents of an Amazon S3 object based on a simple structured query language (SQL) statement. In the request, along with the SQL expressi...


SelectObjectContentCommand | @aws-sdk/client-s3
Documentation for @aws-sdk/client-s3
CSVInput | S3 Client - AWS SDK for JavaScript v3
Documentation for S3 Client - AWS SDK for JavaScript v3
CSV package for Node.js version 6
Version 6 of the package for Node.js is released along its sub projects. Here are the latest versions: version , latest version was NPM version , latest version...
AWS S3 Select and Node.Js
How to use AWS S3 Select and Node.Js