Get CSV object in S3 from Lambda(node.js) with SelectObjectContentCommand – AWS SDK for JavaScript v3
It seems that s3 select can be used in Lambda (node.js). s3 select enables processing of csv data at the time of acquisition.
![](https://confrage.jp/wp-content/uploads/cocoon-resources/blog-card-cache/b01aa5aee99e7f5257b0aadf1baeeafb.png)
Introducing support for Amazon S3 Select in the AWS SDK for JavaScript | Amazon Web Services
We’re excited to announce support for the Amazon Simple Storage Service (Amazon S3) selectObjectContent API with event s...
The CSV placed in S3 should be a CSV of data from the first line as shown below.
項目 | 値 |
---|---|
character code | utf-8 |
newline character | CRLF |
Install the npm module.
npm i @aws-sdk/client-s3 csv-parse
Lambda(node.js)
Get s3 CSV object, s3 select, and make it into an array in the csv module.
import {S3Client, SelectObjectContentCommand} from '@aws-sdk/client-s3' import {parse} from 'csv-parse/sync' export async function handler(event, context) { const input = { Bucket: 'bucket name', Key: 'tmp/testdata.csv', ExpressionType: 'SQL', Expression: 'SELECT s._1,s._2 FROM S3Object s', // SQL to get only columns 1 and 2 InputSerialization: { CSV: { FileHeaderInfo: 'NONE', // headline-less RecordDelimiter: '\r\n', FieldDelimiter: ',' // comma-separated } }, OutputSerialization: { CSV: {} } } const client = new S3Client({ region: 'ap-northeast-1' }) const command = new SelectObjectContentCommand(input) const data = await client.send(command) const csv = parse(await streamToCsv(data.Payload)) console.log(csv) // [ [ 'aaa', 'bbb' ], [ 'ddd', 'eee' ] ] const response = { statusCode: 200, body: JSON.stringify('Hello from Lambda!'), } return response } async function streamToCsv(generator) { const chunks = [] for await (const value of generator) { if (value.Records) { chunks.push(value.Records.Payload) } } return Buffer.concat(chunks).toString('utf8') }
It is troublesome because the Payload is returned in SelectObjectContentEventStream.
InvalidTextEncoding: UTF-8 encoding is required.
It seems that only UTF-8 files are supported, and SJIS will generate an “InvalidTextEncoding: UTF-8 encoding is required.
参考サイト
SelectObjectContentCommand | @aws-sdk/client-s3
Documentation for @aws-sdk/client-s3
CSVInput | S3 Client - AWS SDK for JavaScript v3
Documentation for S3 Client - AWS SDK for JavaScript v3
![](https://confrage.jp/wp-content/uploads/cocoon-resources/blog-card-cache/cf53b366bb2fe2062f9222b209759747.png)
CSV package for Node.js version 6
Version 6 of the package for Node.js is released along its sub projects. Here are the latest versions: version , latest ...
AWS S3 Select and Node.Js
How to use AWS S3 Select and Node.Js
![株式会社CONFRAGE](https://i0.wp.com/confrage.jp/wp-content/uploads/2018/08/aaaaaaaaa-1.jpg?resize=100%2C100&ssl=1)
コメント