Cloudant DB๋ฅผ ์ด์šฉํ•œ Watson IoT platform Historical Data ๊ด€๋ฆฌ

1. Cloudant ์ดํ•ดํ•˜๊ธฐ
2. Design document๋ฅผ ์‘์šฉํ•œ CSV export
3. Watson IoT platform ๊ณผ Cloudant ์—ฐ๋™ํ•˜๊ธฐ

 

์ด 3๊ฐœ์˜ ๊ณผ์ •์œผ๋กœ ์ด๋ฃจ์–ด์ง„ ์‹ค์Šต์˜ ์ฒซ๋ฒˆ์งธ, Cloudant์— ๋Œ€ํ•ด ์•Œ์•„ ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

[Cloudant ์ดํ•ดํ•˜๊ธฐ]

Cloudant DB๋Š” Apache ์žฌ๋‹จ ์˜ opensource project์ธ CouchDB๋ฅผ base๋กœ ๋งŒ๋“ค์–ด์ง„ IBM document base No-sql db ์ž…๋‹ˆ๋‹ค.
IBM Cloud PaaS ์—์„œ๋„ ์ œ๊ณต๋˜๊ณ  , Local ์„ค์น˜๋„ ๊ฐ€๋Šฅ ํ•ฉ๋‹ˆ๋‹ค.

<Local version ์ •๋ณด >
<IBM Cloud PaaS version ์ •๋ณด>

local version์€ license๋ฅผ ๊ตฌ๋งคํ•ด์•ผ ํ•˜๊ณ , cloud PaaS version์€ ์‚ฌ์šฉ๋Ÿ‰์— ๋”ฐ๋ผ ๋น„์šฉ์„ ๋ƒ…๋‹ˆ๋‹ค.

์˜ˆ๋ฅผ๋“ค๋ฉด lookup request 1000๋ฒˆ์€ XXX์› ์ด๋Ÿฐ ์‹์ž…๋‹ˆ๋‹ค.
๋ฌด๋ฃŒ๋กœ ์–ด๋Š์ •๋„๊นŒ์ง€๋Š” ์ด์šฉ๊ฐ€๋Šฅ ํ•˜๋‹ˆ ์‹ค์Šต์„ ํ•˜๊ธฐ์— ์ถฉ๋ถ„ ํ•ฉ๋‹ˆ๋‹ค.
๋ณธ ์‹ค์Šต์—์„œ๋Š” cloud version์„ ์ด์šฉํ•ด๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.
<์ž์„ธํ•œ pricing ์ •๋ณด>

์œ„์—์„œ cloudant ๋ฅผ document base db ๋ผ ํ‘œํ˜„ ํ•˜์˜€๋Š”๋ฐ,์—ฌ๊ธฐ์„œ ์˜๋ฏธํ•˜๋Š” document๊ฐ€ ms word, excel ๊ฐ™์€ file์„ ๋งํ•˜๋Š”๊ฒŒ ์•„๋‹ˆ๋ผ json, xml,yaml ๊ณผ ๊ฐ™์ด ๊ตฌ์กฐํ™”๋œ ๋ฐ์ดํ„ฐ ํƒ€์ž…์„ ๋งํ•ฉ๋‹ˆ๋‹ค.
RDB์—์„œ๋Š” ๋„ˆ๋ฌด๋‚˜ ๋‹น์—ฐํ•˜๊ฒŒ ์‚ฌ์šฉํ•˜๋˜, order by, join, group by, range search, index ๋“ฑ์„ no sql์€ ์ง€์›ํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. RDB ์™€ ์กฐ๊ธˆ ๋‹ค๋ฅธ ํ˜•ํƒœ๋ผ์„œ ์ฒ˜์Œ์— ์‚ฌ์šฉ์ด ์ข€ ์–ด์ƒ‰ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
ํ•˜์ง€๋งŒ ๋Œ€์šฉ๋Ÿ‰ ๋น„๊ด€๊ณ„ํ˜• data๋ฅผ ๋‹ค๋ฃจ๋Š”๋ฐ ์ ํ•ฉํ•œ db์ด๋‹ˆ ์ž˜ ํ™œ์šฉํ•˜๋ฉด ํ›Œ๋ฅญํ•œ performance๋ฅผ ๊ฒฝํ—˜ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

๋งค์šฐ ํฐ ์žฅ์  ์ค‘ ํ•˜๋‚˜๋Š” http ๋ฐฉ์‹์˜ rest api๋ฅผ ์ œ๊ณตํ•œ๋‹ค๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.
๊ณ ์„ฑ๋Šฅ์„ ์š”ํ•˜๋Š” enterprise์—์„  rest ๋ฐฉ์‹์ด ๋ถ€์ ์ ˆ ํ•  ์ˆ˜ ์žˆ์ง€๋งŒ ์ผ๋ฐ˜์ ์ธ ์ค‘ ์†Œ ๊ทœ๋ชจ์˜ transaction์—๋Š” ๊ฐœ๋ฐœ์— ์šฉ์ดํ•˜๊ณ  ํŽธํ•ฉ๋‹ˆ๋‹ค. WEB ๊ณผ Mobile transaction์— ์ ํ•ฉํ•˜๋‹ค๊ณ  ์„ค๋ช… ํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค.
์š”์ฆ˜ ๋Œ€๋ถ€๋ถ„์˜ application client๋ฅผ rest apiํ˜•ํƒœ๋กœ ๊ฐœ๋ฐœํ•˜๋Š” ๊ฒƒ์„ ๊ฐ์•ˆํ•˜๋ฉด rest api์„œ๋ฒ„๋ฅผ ๋”ฐ๋กœ ์šด์˜ํ•˜์ง€ ์•Š์•„๋„ ๋˜๋Š” ๋ถ€๋ถ„์€ ํฐ ์žฅ์  ์ž…๋‹ˆ๋‹ค..

[Cloudant access ํ•˜๊ธฐ]

Cloudant ์— request๋ฅผ ํ•˜๋Š” ๋ฐฉ์‹์€ ๋Œ€ํ‘œ์ ์œผ๋กœ http ๋ฐฉ์‹ ๊ณผ Client library(java,node.js,python) ๋ฅผ ์‚ฌ์šฉํ•˜๋Š” ๋ฐฉ๋ฒ•์ด ์žˆ์Šต๋‹ˆ๋‹ค. http ๋ฐฉ์‹์„ ์ด์šฉํ•˜๋ฉด ํ™˜๊ฒฝ์— ์˜ํ–ฅ์„ ๋ฐ›์ง€ ์•Š๊ณ  ๋ชจ๋“  platform๊ณผ ๋ชจ๋“  language์—์„œ ์‚ฌ์šฉ ๊ฐ€๋Šฅ ํ•ฉ๋‹ˆ๋‹ค.
๋ณธ ์‹ค์Šต์—์„œ๋Š” http ๋ฐฉ์‹์„ ์‚ฌ์šฉํ•˜๋„๋ก ํ•˜๊ฒ ์Šต๋‹ˆ๋‹ค.

๊ธฐ๋ณธ ๋ชจ์–‘:

curl https://$USERNAME:$PASSWORD@$ACCOUNT.cloudant.com/query-demo

database ์ƒ์„ฑ:

curl https://$ACCOUNT.cloudant.com/query-demo -X PUT

doc ์ƒ์„ฑ:

curl https://$ACCOUNT.cloudant.com/query-demo/ -X POST -H “Content-Type: application/json” -dย  { “_id”: “doc1”, “firstname”: “Sally”, “lastname”: “Brown”, “age”: 16, “location”: “New York City, NY”}

์—ฌ๊ธฐ์„œ doc ์ด๋ผ ํ•จ์€ rdb์—์„œ row ๋กœ ๋ณด๋ฉด ๋ฉ๋‹ˆ๋‹ค. data ํ•˜๋‚˜์˜ ๋ฌถ์Œ ์ž…๋‹ˆ๋‹ค. json ํ˜•ํƒœ๋ฅผ ์ทจํ•˜๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค.

bulk ๋กœ doc ์ƒ์„ฑ:

curl https://$ACCOUNT.cloudant.com/query-demo/_bulk_docs -X POST -H “Content-Type: application/json” -d @bulkcreate.dat

json ํ˜•ํƒœ์˜ doc ์„ file๋กœ ์ €์žฅํ•œ ๋’ค ์ด๋ฅผ ์ฝ์–ด์™€ ์—ฌ๋Ÿฌ๊ฐœ์˜ doc์„ ํ•œ๋ฒˆ์— ์ƒ์„ฑํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.

[Cloudant View ์ดํ•ดํ•˜๊ธฐ]

Cloudant๋ฅผ ์ž˜ ํ™œ์šฉํ•˜๊ธฐ ์œ„ํ•ด์„œ๋Š” View๋ฅผ ์ž˜ ํ™œ์šฉํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค. View๋Š” RDB์—์„œ์˜ view์™€ ๋น„์Šทํ•œ ๊ฐœ๋…์„ ๊ฐ–๊ณ  ์žˆ์Šต๋‹ˆ๋‹ค. ์‚ฌ์ „์— ์ •์˜๋œ query๋ฅผ db์— ์ €์žฅํ•ด ๋‘๊ณ  ์ด์— ๋Œ€ํ•œ ๊ฒฐ๊ณผ๋ฅผ ์ œ๊ณต ํ•ฉ๋‹ˆ๋‹ค.
์‹ค์ œ data๋ฅผ ์ฝ์–ด์™€์„œ ์ •์˜๋œ ๊ทœ์น™์— ๋”ฐ๋ผ result set์„ ๋งŒ๋“ค์–ด return ํ•ฉ๋‹ˆ๋‹ค.
cloudant view๋Š” map-reduce๋ฐฉ์‹์œผ๋กœ ๋™์ž‘ ํ•ฉ๋‹ˆ๋‹ค.

map ์ด๋ž€ key, value๋กœ data๋ฅผ mappingํ•˜๋Š” ๊ฒƒ์„ ๋งํ•ฉ๋‹ˆ๋‹ค.
์•„๋ž˜ ์ด๋ฏธ์ง€์—์„œ ๋ณด๋Š” ๋ฐ”์™€ ๊ฐ™์ด 5๊ฐœ์˜ doc ์ด ์žˆ๊ณ  map function์„ ํ†ตํ•ด make ๊ฐ’์„ key ๋กœ ํ•˜๊ณ  value๊ฐ€ 1์ธ map result ๊ฐ€ ๋งŒ๋“ค์–ด ์ง‘๋‹ˆ๋‹ค.

<์ด๋ฏธ์ง€ ์ถœ์ฒ˜: http://yougome.tistory.com/m/392?category=510501>

map function ์ธ emit์˜ ์ฒซ๋ฒˆ์งธ parameter (key ๊ฐ€ ๋˜๋Š” field) ๋Š” indexing ๋˜๋Š” field ์ž…๋‹ˆ๋‹ค.
์‚ฌ์šฉ์ž๋Š” ํ•ด๋‹น key๊ฐ’์„ ์ด์šฉํ•˜์—ฌ query ํ•  ์ˆ˜ ์žˆ๊ณ , key ๊ฐ’์„ ๊ธฐ์ค€์œผ๋กœ ordering ๋ฉ๋‹ˆ๋‹ค.
complex ๊ฐ’์œผ๋กœ sorting์„ ํ•˜๊ณ  ์‹ถ์œผ๋ฉด key๋ฅผ multi๋กœ ์ฃผ๋ฉด ๋ฉ๋‹ˆ๋‹ค. ex)[year, month, day]

reduce๋ž€ map๋œ ๊ฒฐ๊ณผ๋ฅผ ๊ฐ–๊ณ  group by ํ•˜์—ฌ ๊ฐ’์„ ๋„์ถœํ•ด ๋‚ด๋Š” ๊ฒƒ์ž…๋‹ˆ๋‹ค.

<์ด๋ฏธ์ง€ ์ถœ์ฒ˜: http://yougome.tistory.com/m/392?category=510501>

reduce function์€ 3๊ฐœ์˜ parameter๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค.
function (keys, values, rereduce) {
return sum(values);
}

๋งŒ์•ฝ ์„ธ๋ฒˆ์งธ parameter์ธ rereduce๊ฐ€ false์ด๋ฉด , keys๋Š” [key,id] ํ˜•ํƒœ์˜ array๊ฐ€ ๋˜๊ณ  ์—ฌ๊ธฐ์„œ key ๊ฐ’์€ map function์—์„œ ์‚ฌ์šฉ๋œ key ๊ฐ€ ๋ฉ๋‹ˆ๋‹ค. id๋Š” ํ•ด๋‹น key ์˜ document id ๊ฐ’์ž…๋‹ˆ๋‹ค.
values๋Š” keys์— ๋Œ€ํ•ด ์ƒ์„ฑ๋œ ๊ฐ’์˜ array๊ฐ€ ๋ฉ๋‹ˆ๋‹ค.
ex) reduce([ [key1,id1], [key2,id2], [key3,id3] ], [value1,value2,value3], false)

rereduce๊ฐ€ true์ด๋ฉด keys ๋Š” null ์ด๊ณ  values๋Š” reduce funtion์˜ ์ด์ „ call๋กœ ๋ถ€ํ„ฐ return ๋œ ๊ฐ’์ž…๋‹ˆ๋‹ค.
ex) reduce(null, [intermediate1,intermediate2,intermediate3], true)

 

built-in reduce function์œผ๋กœ _count, _stats, _sum ์ด ์žˆ์Šต๋‹ˆ๋‹ค.

 

view๋ฅผ ์ƒ์„ฑํ•˜๋Š” ๋ฐฉ๋ฒ•์€ document๋ฅผ ์ƒ์„ฑํ•˜๋Š” ๋ฐฉ์‹๊ณผ ๋™์ผ ํ•ฉ๋‹ˆ๋‹ค. ๋‹ค๋งŒ view๋Š” design document๋กœ ๋ถ„๋ฆฌ ๋˜๋ฏ€๋กœ endpoint๊ฐ€ ์กฐ๊ธˆ ๋‹ค๋ฆ…๋‹ˆ๋‹ค.

curl -X PUT https://$ACCOUNT:$PASSWORD@$ACCOUNT.cloudant.com/$DATABASE/_design/training
–data-binary @view.def

view.def ํŒŒ์ผ์— view ์˜ function code๊ฐ€ ์ •์˜๋˜๋ฉด ๋ฉ๋‹ˆ๋‹ค. (ํ™•์žฅ์ž๊ฐ€ ๊ผญ def์ผ ํ•„์š”๋Š” ์—†์Šต๋‹ˆ๋‹ค.)
training์ด๋ผ๋Š” design document๊ฐ€ ์ƒ์„ฑ ๋  ๊ฒƒ์ด๊ณ  ๊ทธ ์•ˆ์— view๊ฐ€ ์ •์˜ ๋  ๊ฒƒ์ž…๋‹ˆ๋‹ค.

 

[Cloudant Design Document ์ดํ•ดํ•˜๊ธฐ]

Design document๋Š” cloudant์—์„œ view, index ๋“ฑ์„ ์ •์˜ํ•˜๋Š” ๋ฌธ์„œ ์ž…๋‹ˆ๋‹ค. json type์œผ๋กœ ์ •์˜ ๋ฉ๋‹ˆ๋‹ค.

Cloudant๋ฅผ ์ œ๋Œ€๋กœ ํ™œ์šฉํ•˜๊ธฐ ์œ„ํ•ด์„œ๋Š” design document๋ฅผ ๋ช…ํ™•ํžˆ ์•Œ์•„์•ผ ํ•ฉ๋‹ˆ๋‹ค. ์กฐ๊ธˆ ๋ณต์žกํ•˜์ง€๋งŒ ๊ผญ ์ดํ•ดํ•˜๊ณ  ๋„˜์–ด๊ฐ€์„ธ์š”.
Design document์˜ ๊ตฌ์กฐ๋Š” ์•„๋ž˜์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค.

  • _id: Design Document ID
  • _rev: Design Document Revision
  • views (optional): MapReduce ์ˆ˜ํ–‰ํ•˜๋Š”
    • viewname : View ์ด๋ฆ„, ์ด๋ฅผ ํ†ตํ•ด view์— ์ ‘๊ทผ ํ•ฉ๋‹ˆ๋‹ค..
      • map: Map Function ์ •์˜ .
      • reduce (optional): Reduce Function ์ •์˜.
    • indexes (optional): search indexes. ๊ฒ€์ƒ‰์„ ์œ„ํ•ด ์‚ฌ์šฉ๋˜๋Š” secondary index,
      • index name: Index ์ด๋ฆ„.
        • analyzer: text๋กœ ๋ถ€ํ„ฐ index term์„ ์–ด๋–ป๊ฒŒ ๊ฐ€์ ธ์˜ฌ์ง€ defineํ•˜๋Š” ์—ญํ• :
          • name: analyzer๋ฅผ ์„ ํƒ ํ•ฉ๋‹ˆ๋‹ค. (standard, email, keyword, simple, whitespace, classic, perfield) ๋“ฑ์„ ์„ ํƒ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
          • stopwords (optional): index๋˜์ง€ ์•Š๊ธธ ์›ํ•˜๋Š” ๋‹จ์–ด๋ฅผ ์ง€์ •ํ•ฉ๋‹ˆ๋‹ค. ์ง€์ •ํ•˜์ง€ ์•Š์œผ๋ฉด default๋กœ ์ง€์ •๋˜์–ด ์žˆ๋Š” ๊ฐ’๋“ค์ด ์‚ฌ์šฉ ๋ฉ๋‹ˆ๋‹ค. [a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, no, not, of, on, or, such, that, the, their, then, there, these, they, this, to, was, will] with]
          • default (for the per field analyzer): fields ์— ์–ธ์–ด๊ฐ€ ํŠน์ •๋˜์ง€ ์•Š์„๊ฒฝ์šฐ ์‚ฌ์šฉ๋  default ์–ธ์–ด.
          • fields (for the per field analyzer): index๋  field์˜ ์–ธ์–ด๋ฅผ ์ง€์ • ํ•ฉ๋‹ˆ๋‹ค.
        • index: index ๋ฅผ handling ํ•˜๋Š” Function ์ •์˜
      • shows (optional): Show function์€ document์˜ ํ˜•ํƒœ๋ฅผ ์ •์˜ํ•ฉ๋‹ˆ๋‹ค.
        • function name: Function definition.
      • lists (optional): List functions ์€ show function๊ณผ ์—ญํ• ์€ ๊ฐ™์€๋ฐ, view function์˜ ๊ฒฐ๊ณผ์— ๋Œ€ํ•ด format์„ ์ง€์ •ํ•ฉ๋‹ˆ๋‹ค.
        • function name: Function definition

์ดํ•ด๋ฅผ ๋•๊ธฐ ์œ„ํ•ด ์‹ค์ œ๋กœ ์‚ฌ์šฉํ•˜๋Š” design document format์„ ์ฒจ๋ถ€ ํ•ฉ๋‹ˆ๋‹ค. ๊ฐ๊ฐ์˜ function ๋ถ€๋ถ„์€ ๋ณต์žกํ•ด์„œ ๋‹ค ํ‘œํ˜„ํ•˜์ง€ ๋ชปํ–ˆ์Šต๋‹ˆ๋‹ค. ์ดํ›„ ๊ณผ์ •์—์„œ ์ „์ฒด code๋ฅผ ๋ณผ ๊ธฐํšŒ๊ฐ€ ์žˆ์œผ๋‹ˆ ๊ทธ๋•Œ ์ž์„ธํžˆ ์•Œ์•„ ๋ณด๊ฒ ์Šต๋‹ˆ๋‹ค.

design document json format ์ฐธ๊ณ :

{
ย  "_id": "_design/iotp",
ย  "_rev": "4-bbf25a2e6be349cebf337e3adbb8af12",
ย  "language": "javascript",
ย  "views": {
ย ย ย  "by-deviceId": {
ย ย ย ย ย  "map":"function(doc){\n
               if(doc.deviceId && doc.timestamp){\n
                  emit([doc.deviceId,doc.timestamp],doc);\n}\n}\n"
ย ย ย  }
ย  },
ย  "lists": {
ย ย ย  "csv": "function(head,req) {\n ......\n};\n"
ย  },
ย  "indexes": {
ย ย ย  "search": {
ย ย ย ย ย  "analyzer": "keyword",
ย ย ย ย ย  "index": "function(doc) {\nย  .......\nย  }\n}"
ย ย ย  }
ย  }
}

ํ† ๋ก  ์ฐธ๊ฐ€

์ด๋ฉ”์ผ์€ ๊ณต๊ฐœ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ํ•„์ˆ˜ ์ž…๋ ฅ์ฐฝ์€ * ๋กœ ํ‘œ์‹œ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค