Get CSV object in S3 from Lambda(node.js) with SelectObjectContentCommand – AWS SDK for JavaScript v3

Get CSV object in S3 from Lambda(node.js) with SelectObjectContentCommand – AWS SDK for JavaScript v3

It seems that s3 select can be used in Lambda (node.js). s3 select enables processing of csv data at the time of acquisition.

Introducing support for Amazon S3 Select in the AWS SDK for JavaScript | Amazon Web Services

We’re excited to announce support for the Amazon Simple Storage Service (Amazon S3) selectObjectContent API with event s...

aws.amazon.com

The CSV placed in S3 should be a CSV of data from the first line as shown below.

2行3列のcsv

Install the npm module.

npm i @aws-sdk/client-s3 csv-parse

Lambda(node.js)

Get s3 CSV object, s3 select, and make it into an array in the csv module.

import {S3Client, SelectObjectContentCommand} from '@aws-sdk/client-s3'
import {parse} from 'csv-parse/sync'
export async function handler(event, context) {
  const input = {
    Bucket: 'bucket name',
    Key: 'tmp/testdata.csv',
    ExpressionType: 'SQL',
    Expression: 'SELECT s._1,s._2 FROM S3Object s', // SQL to get only columns 1 and 2
    InputSerialization: {
      CSV: {
        FileHeaderInfo: 'NONE', // headline-less
        RecordDelimiter: '\r\n',
        FieldDelimiter: ',' // comma-separated
      }
    },
    OutputSerialization: {
      CSV: {}
    }
  }
  const client = new S3Client({
    region: 'ap-northeast-1'
  })
  const command = new SelectObjectContentCommand(input)
  const data = await client.send(command)
  const csv = parse(await streamToCsv(data.Payload))
  console.log(csv) // [ [ 'aaa', 'bbb' ], [ 'ddd', 'eee' ] ]
  
  const response = {
    statusCode: 200,
    body: JSON.stringify('Hello from Lambda!'),
  }
  return response
}

async function streamToCsv(generator) {
  const chunks = []
  for await (const value of generator) {
    if (value.Records) {
      chunks.push(value.Records.Payload)
    }
  }
  return Buffer.concat(chunks).toString('utf8')
}

It is troublesome because the Payload is returned in SelectObjectContentEventStream.

InvalidTextEncoding: UTF-8 encoding is required.

It seems that only UTF-8 files are supported, and SJIS will generate an “InvalidTextEncoding: UTF-8 encoding is required.

参考サイト

SelectObjectContentCommand | @aws-sdk/client-s3

Documentation for @aws-sdk/client-s3

docs.aws.amazon.com

CSVInput | S3 Client - AWS SDK for JavaScript v3

Documentation for S3 Client - AWS SDK for JavaScript v3

docs.aws.amazon.com

CSV package for Node.js version 6

Version 6 of the package for Node.js is released along its sub projects. Here are the latest versions: version , latest ...

www.adaltas.com

AWS S3 Select and Node.Js

How to use AWS S3 Select and Node.Js

blog.salvatorecozzubo.com

高橋